Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views

Hugging Face Daily Papers Papers
3d-reconstruction instance-segmentation feed-forward multi-view differentiable-rendering scene-decomposition tokenization

Summary

This paper proposes a feed-forward framework that decomposes 3D scenes into instance-structured token groups from unposed multi-view images, enabling direct object-level reconstruction, segmentation, and manipulation without 3D annotations.

A 3D scene is understood through its objects, not the primitives that compose them. Yet feed-forward reconstruction methods output dense, unstructured sets of points or Gaussians, leaving object-level structure to be recovered after the fact. We propose a feed-forward framework that decomposes a scene into instance-structured 3D token groups directly from unposed multi-view images -- compact object-centric units from which reconstruction, segmentation, and manipulation all follow. Each token group pairs an instance token capturing entity-level identity with anchor tokens that encode local geometry and appearance, which are decoded into a set of 3D Gaussians. This two-level factorization decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product. The token groups are learned through differentiable rendering with joint reconstruction and segmentation supervision, requiring no 3D annotations. Our feed-forward model surpasses per-scene optimization baselines in class-agnostic instance segmentation while remaining competitive in novel view synthesis. Beyond these metrics, the same token groups directly unlock instance-level scene editing -- removing, translating, or inserting objects by operating on their groups -- as well as efficient open-vocabulary 3D instance retrieval, where retrieval complexity scales with the number of instances rather than primitives.
Original Article
View Cached Full Text

Cached at: 07/01/26, 11:42 AM

Paper page - Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views

Source: https://huggingface.co/papers/2606.29513

Abstract

A feed-forward framework decomposes 3D scenes into instance-structured token groups from multi-view images, enabling direct object-level reconstruction, segmentation, and manipulation without 3D annotations.

A 3D scene is understood through its objects, not the primitives that compose them. Yet feed-forward reconstruction methods output dense, unstructured sets of points or Gaussians, leaving object-level structure to be recovered after the fact. We propose afeed-forward frameworkthat decomposes a scene into instance-structured 3D token groups directly fromunposed multi-view images-- compact object-centric units from which reconstruction, segmentation, and manipulation all follow. Each token group pairs an instance token capturing entity-level identity with anchor tokens that encode local geometry and appearance, which are decoded into a set of3D Gaussians. This two-level factorization decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product. The token groups are learned throughdifferentiable renderingwithjoint reconstruction and segmentation supervision, requiring no 3D annotations. Our feed-forward model surpasses per-scene optimization baselines inclass-agnostic instance segmentationwhile remaining competitive innovel view synthesis. Beyond these metrics, the same token groups directly unlockinstance-level scene editing-- removing, translating, or inserting objects by operating on their groups -- as well as efficientopen-vocabulary 3D instance retrieval, where retrieval complexity scales with the number of instances rather than primitives.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.29513

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.29513 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.29513 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.29513 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Scene Abstraction for Lexical Semantics: Structured Representations of Situated Meaning

arXiv cs.CL

This paper proposes Scene Abstraction, a framework for constructing structured representations of the interpretive scenes that words evoke in context, using few-shot prompting of large language models. The authors introduce COCA-Scenes, a dataset of 520 usage instances, and provide empirical evidence that scenes are reliably identifiable and align better with human interpretation than alternatives.

TriSplat: Simulation-Ready Feed-Forward 3D Scene Reconstruction

Hugging Face Daily Papers

TriSplat is a feed-forward 3D reconstruction network that uses oriented triangle primitives to directly generate simulation-ready meshes from single images, bypassing expensive post-processing steps. It achieves geometry-faithful reconstructions while maintaining competitive novel-view rendering quality.

Direct 3D-Aware Object Insertion via Decomposed Visual Proxies

Hugging Face Daily Papers

This paper introduces DIRECT, a framework for pose-controllable 3D-aware object insertion that decomposes conditions into appearance, geometry, and context guidance to achieve high-fidelity compositing with explicit 3D pose control.

4D Human-Scene Reconstruction from Low-Overlap Captures

Hugging Face Daily Papers

Proposes StudioRecon, a method for 4D human-scene reconstruction from low-overlap camera captures, using separate priors for backgrounds (video diffusion) and humans (SMPL), with a recursive enhancement module. Accepted to SIGGRAPH 2026, outperforming prior methods by +1.5 to +5.0 dB PSNR.

From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

Hugging Face Daily Papers

This paper introduces a multimodal image fusion method that uses a 1D token interface from a pretrained image tokenizer to enhance global appearance coherence while preserving local details through selective token editing (STE). Experiments on four benchmarks show state-of-the-art performance in both global coherence and local fidelity.