Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Hugging Face Daily Papers Papers
multimodal-llm 3d-spatial-reasoning gaussian-splatting cross-view-learning learnable-tokens photometric-reconstruction

Summary

Imagine3D-LLM proposes teaching multimodal LLMs to imagine a compact 3D representation of a scene (decoded from learnable summary tokens as 3D Gaussian Splatting via photometric reconstruction loss) before answering multi-view spatial reasoning questions, outperforming prior 3D-aware MLLM approaches on multiple benchmarks.

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:21 AM

Paper page - Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Source: https://huggingface.co/papers/2609.38177 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Reasoningaboutthe3Dworldfrommulti-viewimagesremainsafundamentalchallengeforMultimodalLargeLanguageModels(MLLMs).WhilemodernMLLMshandlesingle-imageinputseffectively,theystruggletointegrateevidenceacrossviewpointsintoacoherent3Dunderstanding.Agrowingbodyofworkattemptstoclosethisgapbyinjecting3DawarenessintoMLLMs,eitherbyboostingfine-grainedpixel-levelcross-viewcorrespondenceorbyfusingfeaturesfrom3Dgeometryfoundationmodels,yetasubstantialgaptohumanreasoningpersists.Inthiswork,werevisithumanspatialreasoning,whichsuggeststhatratherthanrelyingonfine-grainedgeometrycues,humansroughlyidentifycommonobjectsacrossviews,infertherelativegeometrybetweenviewpoints,andassembleacoarse3Dlayoutofthescene.Inspiredbythisprocess,weintroduceImagine3D-LLM,anMLLMthatlearnstoassembleasimilarcompact3Drepresentationofthesceneandconditionsitsansweronthisrepresentation.Concretely,weappendasmallsetoflearnablesummarytokensaftertheimagetokens,decodethemintoacompact3DGaussianSplattingrepresentationsupervisedbyaphotometricreconstructionloss,andtrainjointlywiththestandardnext-tokenpredictionobjective.Notably,althoughonlythesummarytokensreceivedirectreconstructionsupervision,thisobjectivealsoinducesstrongercross-framecorrespondencewithintheLLM’sunderlyingimagefeatures,suggestingthatlearningtoreconstructpropagates3D-awaresignalsthroughoutthemodel.Asaresult,Imagine3D-LLMconsistentlyoutperformspriorapproachesacrossmultiplespatialreasoningand3Dunderstandingbenchmarks,suggestingthatimaginingthescenecanbemoreeffectivethanbeingtolditspixel-wisegeometry.

View arXiv pageView PDFProject pageGitHub16Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.38177 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.38177 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.38177 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

LLM2R: From Large Language Models to 3D Reassemblers

Hugging Face Daily Papers

LLM2R casts 3D part assembly as sequence modeling with a pretrained decoder-only LLM, using geometric tokens and learnable summary tokens to predict all part poses in a single non-autoregressive pass, achieving state-of-the-art accuracy on PartNet, PartNeXt, and Breaking Bad benchmarks.

Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching

Hugging Face Daily Papers

This paper introduces ReasonMatch-Bench, a benchmark for wide-baseline matching in multimodal LLMs, and proposes Dynamic Correspondence Reinforcement Learning (DCRL) to improve spatial reasoning. Experiments show significant gains on the benchmark while maintaining general performance.