Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
Summary
Stream3D-VLM is an online 3D vision-language model that enables real-time spatial understanding from streaming video by incrementally integrating geometry priors and using geometry-adaptive voxel compression, outperforming existing models on 3D spatial understanding tasks.
View Cached Full Text
Cached at: 06/08/26, 07:14 AM
Paper page - Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
Source: https://huggingface.co/papers/2606.06891
Abstract
An online 3D vision-language model enables real-time spatial understanding from streaming video using autoregressive control modeling and efficient visual token compression.
Despite advances in 3D scene understanding, existing3D Large Multimodal Modelsoperate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts anautoregressive streaming control modelingbased on the LLM’snext-token prediction objectiveto learn when to respond, and employs a lightweightVisual-Spatial Feature Integration(VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-playGeometry-Adaptive Voxel Compression(GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M onlinespatio-temporal 3D QA pairsand establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/
View arXiv pageView PDFProject pageGitHub3Add to collection
Community
Stream3D-VLM is an online 3D vision-language model that supports real-time spatial understanding and interaction directly from streaming video. By incrementally integrating geometry priors and employing geometry-adaptive voxel compression, our approach enables efficient and continuous 3D scene comprehension without requiring offline processing or complete scene observations.
Upload images, audio, and videos by dragging in the text input, pasting, orclicking here.
Tap or paste here to upload images
Get this paper in your agent:
hf papers read 2606\.06891
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### JonnyYu828/Stream3D-VLM-4B Image-Text-to-Text• 5B• Updatedabout 3 hours ago • 56 • 2
Datasets citing this paper2
#### JonnyYu828/Stream3D-1M-Dataset Preview• Updatedabout 3 hours ago • 32 • 1 #### JonnyYu828/Stream3D-Bench Viewer• Updatedabout 3 hours ago • 10k • 18 • 1
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.06891 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
This paper proposes GASP, a framework that injects geometric priors into vision-language models via deep supervision with contrastive and depth consistency losses, achieving significant improvements on 3D spatial reasoning benchmarks without using 3D VQA data.
Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models
GeoVR enhances multimodal large language models with 3D awareness by restructuring their semantic latent space through geometric knowledge distillation from 3D foundation models using multiple geometric targets.
MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
MetaSpatial is a reinforcement learning framework that enhances 3D spatial reasoning in vision-language models, enabling coherent and physically plausible 3D scene generation without hard-coded optimizations.
Geometric Context Transformer for Streaming 3D Reconstruction
Introduces LingBot-Map, a feed-forward 3D foundation model for streaming 3D reconstruction using a geometric context transformer architecture that achieves stable real-time performance at 20 FPS.
OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
OVO-S-Bench introduces a comprehensive human-annotated benchmark of 1,680 questions across 348 videos to evaluate streaming spatial intelligence in multimodal LLMs, revealing that even the best model (Gemini-3.1-Pro) trails human experts by 27 points. The benchmark exposes key limitations including allocentric mapping as a major bottleneck and chain-of-thought reasoning amplifying spatial errors.