Metraipolis › archive › dc_fe0194f652b0
SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding — arXiv:2608.05137
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.05137
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:48:20.392Z
- SHA-256 of the archived text
26094dd59d53418beac7f8d7c4bc2f1c6b2d0ab40f5fbcbeae33a3bfae6d8e30- Stable URL of this page
https://metraipolis.com/documento/dc_fe0194f652b0
How to cite
SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding — arXiv:2608.05137 — arXiv (en), https://arxiv.org/abs/2608.05137 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fe0194f652b0 (Metraipolis archive dc_fe0194f652b0, sha256 26094dd59d53418beac7f8d7c4bc2f1c6b2d0ab40f5fbcbeae33a3bfae6d8e30)
BibTeX
@misc{dcfe0194f652b0,
title = {SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding — arXiv:2608.05137},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.05137},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fe0194f652b0; sha256 26094dd59d53418beac7f8d7c4bc2f1c6b2d0ab40f5fbcbeae33a3bfae6d8e30},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05137. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTing (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: this https URL . Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-06 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2608.05137 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:37:40.747Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).