SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Hugging Face Daily Papers Papers
3d-scene-understanding multimodal-llm modality-orchestration embodied-intelligence arxiv

Summary

SmartMage is a unified multimodal LLM that dynamically orchestrates visual and geometric modalities for query-dependent 3D scene understanding, achieving state-of-the-art results across five benchmarks.

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:55 AM

Paper page - SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Source: https://huggingface.co/papers/2608.05137

Abstract

Understanding3Dscenesisfundamentaltoembodiedintelligence,requiringjointreasoningoverheterogeneousinformationfrommultiplemodalities,includingvisualandgeometriccues.However,therelevanceofthesemodalitiesoftenvariesacrossqueries.ExistingMultimodalLargeLanguageModels(MLLMs)typicallyrelyonfixedmodalitycombinations,overlookingquery-dependentmodalityneeds.Sucharigiddesigncanintroducesemanticnoisefromirrelevantmodalitieswhileunderutilizingmoreinformativeones,leadingtowastedcomputationanddilutedreasoning.Toaddressthesechallenges,thispaperproposesSmartMage,aunifiedMLLMthatdynamicallyorchestratesheterogeneousmodalitiesforsemantic-aware3Dsceneunderstanding.Specifically,SmartMageincorporates:(1)aSemantic-guidedModalityAdaptiveRouTng(SMART)modulethatselectstask-relevantmodalitiesusingsemanticpriors,text-modalityalignment,andmodalityquality;and(2)aModality-AwareGatingExpert(MAGE)modulethatleveragesmodalitypriorstoguideexpertactivation,fosteringadaptivespecializationinmultimodalreasoning.Empirically,SmartMageachievesstate-of-the-artperformanceacrossfive3Dsceneunderstandingbenchmarks,andattainscompetitiveresultsonRGB-onlyvideounderstandingbenchmarks.InourdiagnosticbenchmarkScanFacet,tasksaredividedintofine-grainedsemanticcategories,enablinganalysisofmodalitycombinationspreferredbyeachsemantictype.Theobservedmodality-semanticpatternsprovidefurtherevidenceofSmartMage’seffectiveness.Projectpage:https://yuecheong.github.io/SmartMage/.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.05137

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.05137 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.05137 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.05137 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Mage (GitHub Repo)

TLDR AI

Microsoft releases Mage, a family of lightweight 4B-parameter multimodal models for visual understanding and generation, including Mage-VL for image/video understanding and Mage-Flow for text-to-image generation and editing, designed for research and deployment on modest hardware.

Mage (2 minute read)

TLDR AI

Microsoft unveils Mage, a family of compact 4B-parameter multimodal models for visual understanding and generation, designed for easy training and deployment on modest hardware while staying competitive with larger open models.

MAGE: Human-Like Macro Placement via Agentic Multimodal Reasoning

arXiv cs.AI

MAGE is a multimodal multi-agent framework that improves macro placement in VLSI physical design by combining structured floorplanning rules, visual checks, and iterative refinement, outperforming commercial placers and human experts on timing metrics.