Metraipolis › archive › dc_2b0d7517db33

Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models — arXiv:2611.05927

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05927
License (as archived)
CC0-1.0
Archived
2026-11-05T18:18:55.985Z
SHA-256 of the archived text
311024957f64eac667a1f9c5b39ee107e0a0f3fbbeef899fe69e14efcf31bb04
Stable URL of this page
https://metraipolis.com/documento/dc_2b0d7517db33

How to cite

Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models — arXiv:2611.05927 — arXiv (en), https://arxiv.org/abs/2611.05927 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b0d7517db33 (Metraipolis archive dc_2b0d7517db33, sha256 311024957f64eac667a1f9c5b39ee107e0a0f3fbbeef899fe69e14efcf31bb04)
BibTeX
@misc{dc2b0d7517db33,
  title = {Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models — arXiv:2611.05927},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05927},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b0d7517db33; sha256 311024957f64eac667a1f9c5b39ee107e0a0f3fbbeef899fe69e14efcf31bb04},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05927. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e.g., 54 GB out of about 58 GB in BF16 for InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B). Expert pruning has emerged as an effective compression strategy, but existing methods aggregate importance uniformly across tokens, ignoring that the same expert may serve different roles for vision versus language processing. To address this, we propose MAEP (Modality-Aware Expert Pruning), a training-free framework that decomposes expert importance by modality in a single forward pass. MAEP introduces (1) Delta-H Output ($\Delta H$), a redistribution-aware metric measuring the MoE output change when an expert is removed, and (2) low-attention vision tokens as a vision-side pruning signal that is dataset-stable: its expert-importance ranking remains stable across different datasets. Text-side importance serves as a preservation signal while vision-side importance serves as a pruning signal, and experts are ranked globally across layers. Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05927 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:13:03.952Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).