Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Hugging Face Daily Papers Papers
3d-generation diffusion-transformers latent-space scene-generation representation-learning 3d-vision

Summary

提出在由2D基础模型导出的高维隐式3D潜在空间中直接进行扩散建模，实现3D-grounded的场景级3D生成，相比2D多视角方法在空间一致性和效率上更具优势，并验证了其在表示与模型容量上的可扩展性。

Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated scenes. In this paper, we propose to perform 3D scene generation directly within a high-dimensional implicit 3D latent space derived from powerful 2D foundation models (e.g., DINOv2, SigLIP2, DA3). Then we tame diffusion transformers to perform diffusion modeling directly in the 3D latent space, enabling 3D-grounded scene generation. In our experiments, we not only demonstrate the advantages of our method over 2D-based approaches in terms of efficiency and spatial consistency, but also comprehensively analyze the impact of 3D latent representation on 3D scene generation performance. Furthermore, we validate the favorable scalability of our method with respect to representation and model capacity.
Original Article
View Cached Full Text

Cached at: 10/21/26, 07:05 AM

Paper page - Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Source: https://huggingface.co/papers/2604.11331

Abstract

Direct diffusion modeling in a 3D latent space improves spatial consistency and efficiency for scene-level 3D generation compared to 2D-based methods.

Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits thespatial consistencyof the generated scenes. In this paper, we propose to perform 3D scene generation directly within a high-dimensional implicit3D latent spacederived from powerful2D foundation models(e.g.,DINOv2,SigLIP2,DA3). Then we tamediffusion transformersto perform diffusion modeling directly in the3D latent space, enabling3D-grounded scene generation. In our experiments, we not only demonstrate the advantages of our method over 2D-based approaches in terms of efficiency andspatial consistency, but also comprehensively analyze the impact of 3D latent representation on 3D scene generation performance. Furthermore, we validate the favorable scalability of our method with respect to representation and model capacity.

View arXiv pageView PDFProject pageGitHub65Add to collection

Get this paper in your agent:

hf papers read 2604\.11331

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### tracywei/3drae Image-to-3D• Updated1 day ago

Datasets citing this paper1

#### tracywei/scannet_3drae

Spaces citing this paper1

Collections including this paper1

Similar Articles

Structured 3D Latents for Scalable and Versatile 3D Generation

Papers with Code Trending

微软提出统一的结构化潜在表示（SLAT），结合 rectified flow transformers 实现高质量、多格式（Radiance Fields、3D Gaussians、网格）的 3D 资产生成，支持文本/图像条件及局部编辑，在 500K 3D 数据集上训练的 2B 参数模型显著超越现有方法。

Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation

Hugging Face Daily Papers

Scene-Designer 提出一个通过设计与批评双重经验学习设计智能的智能体框架，用于生成完整、连贯且可用的 3D 室内场景，并提出 Critic 引导的 Scene-GRPO 方法，在物理有效性、功能正确性和布局连贯性上取得更优性能。