MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
Summary
MetaSpatial is a reinforcement learning framework that enhances 3D spatial reasoning in vision-language models, enabling coherent and physically plausible 3D scene generation without hard-coded optimizations.
View Cached Full Text
Cached at: 07/05/26, 10:32 AM
Paper page - MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
Source: https://huggingface.co/papers/2503.18470 Published on Mar 24, 2025
Abstract
MetaSpatial is a reinforcement learning framework that enhances 3D spatial reasoning in vision-language models, enabling realistic and coherent 3D scene generation through iterative refinement and physics-aware constraints.
We present MetaSpatial, the firstreinforcement learning(RL)-based framework designed to enhance3D spatial reasoninginvision-language models(VLMs), enablingreal-time 3D scene generationwithout the need for hard-coded optimizations. MetaSpatial addresses two core challenges: (i) the lack of internalized3D spatial reasoningin VLMs, which limits their ability to generate realistic layouts, and (ii) the inefficiency of traditional supervised fine-tuning (SFT) for layout generation tasks, as perfect ground truth annotations are unavailable. Our key innovation is amulti-turn RL-based optimization mechanism that integratesphysics-aware constraintsand rendered image evaluations, ensuring generated 3D layouts are coherent, physically plausible, and aesthetically consistent. Methodologically, MetaSpatial introduces an adaptive, iterative reasoning process, where the VLM refines spatial arrangements over multiple turns by analyzing rendered outputs, improving scene coherence progressively.Empirical evaluationsdemonstrate that MetaSpatial significantly enhances the spatial consistency and formatting stability of various scale models. Post-training, object placements are more realistic, aligned, and functionally coherent, validating the effectiveness of RL for3D spatial reasoninginmetaverse,AR/VR,digital twins, and game development applications. Our code, data, and training pipeline are publicly available at https://github.com/PzySeere/MetaSpatial.
View arXiv pageView PDFProject pageGitHub322Add to collection
Get this paper in your agent:
hf papers read 2503\.18470
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2503.18470 in a model README.md to link it from this page.
Datasets citing this paper3
#### johnschaefer/EasyR1-qwen3vl-rl Updated22 days ago • 577 #### Yuting6/ttrl Viewer• UpdatedApr 30, 2025 • 6 • 434 #### zhenyupan/3d_layout_reasoning Viewer• UpdatedMar 25, 2025 • 100 • 117
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2503.18470 in a Space README.md to link it from this page.
Collections including this paper2
Similar Articles
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
SpatialAct is a new simulator-grounded benchmark that probes whether VLM agents can perform coherent spatial reasoning and translate it into actions in 3D environments across multi-turn feedback settings. Experiments reveal a significant reasoning-to-action gap, with current VLMs struggling to maintain spatial beliefs and produce reliable actions despite performing well on isolated reasoning tasks.
Reinforcing Dual-Path Reasoning in Spatial Vision Language Models
This paper introduces SR-REAL, a unified framework for spatial vision-language models that combines linguistic deduction and 3D geometric reasoning via reinforcement learning, enabling robust multi-step spatial reasoning across diverse tasks.
Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators
The paper proposes Astra, an agentic spatial reasoning framework that couples a reinforcement learning-trained VLM policy with a world simulator to generate novel-view observations for improved spatial reasoning in Vision-Language Models.
SVoT: State-aware Visualization-of-Thought for Spatial Reasoning via Reinforcement Learning
The paper proposes SVoT, a reinforcement learning framework that generates interleaved, verifiable intermediate states and visualizations for multi-hop spatial reasoning in MLLMs, achieving significant accuracy gains on new benchmarks involving multi-object interactions and numerical reasoning.
Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
Stream3D-VLM is an online 3D vision-language model that enables real-time spatial understanding from streaming video by incrementally integrating geometry priors and using geometry-adaptive voxel compression, outperforming existing models on 3D spatial understanding tasks.