ACM Multimedia 2026

SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Yue Zhang1 Yingzhao Jian1 Yunqiu Xu1 Xiaoxiao Sun2 Hehe Fan1,✉
1 Zhejiang University, Hangzhou, Zhejiang, China 2 Stanford University, Stanford, California, USA
✉ Corresponding author
arXiv Paper DOI Code Hugging Face
Motivation of SmartMage: different questions prefer different 3D modalities, while SmartMage adaptively selects modalities and experts.
Different questions exhibit inherent modality preferences. SmartMage replaces fixed modality combinations with semantic-guided adaptive selection and modality-aware expert routing.
Overview

Abstract

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous visual and geometric cues. Yet the relevance of these modalities varies across queries, while existing Multimodal Large Language Models typically rely on fixed modality combinations. SmartMage is a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. It combines a Semantic-guided Modality Adaptive RouTing (SMART) module, which selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality, with a Modality-Aware Gating Expert (MAGE) module, which uses modality priors to guide expert activation and foster adaptive specialization. SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks and remains competitive on RGB-only video benchmarks. ScanFacet further enables fine-grained analysis of modality preferences across semantic question types.

Approach

Global-to-local modality orchestration

SmartMage first decides which modalities should participate in reasoning, then assigns multimodal tokens to appropriate experts inside the language model.

SMART: adaptive modality selection

A global scheduler jointly considers semantic priors, text-modality similarity, and modality quality to retain RGB as the primary modality and select an instruction-dependent subset of BEV, depth, point cloud, and voxel inputs.

MAGE: modality-aware experts

A modality-aware expert speculation module supplies soft modality-expert priors to a sparse MoE-based LLM, encouraging structured token routing and modality-specialized expert behavior.

Architecture of SmartMage with feature extraction, SMART modality selection, and MAGE expert routing.
SmartMage encodes RGB, depth, BEV, point cloud, and voxel inputs in a unified space; SMART selects complementary modalities and MAGE performs modality-aware expert routing.
Experiments

Strong performance across five 3D benchmarks

SmartMage is evaluated on 3D question answering, dense captioning, and visual grounding, while also supporting interpretable task-dependent modality selection.

32.6ScanQA
EM@1
66.8SQA3D
EM@1
88.7Scan2Cap
CIDEr@0.5
59.5ScanRefer
Acc@0.5
60.7Multi3DRefer
F1@0.5
Qualitative SmartMage results for object identification, spatial reasoning, color recognition, counting, localization, and geometry understanding.
Qualitative examples show that the model selects different modality combinations for object identification, spatial reasoning, color recognition, counting, localization, and geometry understanding.
Heatmap of modality preferences across semantic question types.
Adaptive modality combinations vary across question semantics on ScanFacet.
Token allocation across experts with modality-aware expert speculation.
MAGE induces differentiated modality-to-expert allocation patterns, revealing modality-aware expert specialization.
Citation

BibTeX

Please cite our ACM Multimedia 2026 paper when using SmartMage in your research.

@inproceedings{zhang2026smartmage,
  title     = {SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding},
  author    = {Zhang, Yue and Jian, Yingzhao and Xu, Yunqiu and Sun, Xiaoxiao and Fan, Hehe},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026},
  doi       = {10.1145/3767308.3835083}
}