Metraipolis › archive › dc_aebae90da23a

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit — arXiv:2609.31631

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31631
License (as archived)
CC0-1.0
Archived
2026-10-07T22:36:01.066Z
SHA-256 of the archived text
cb4d6d4ec8e1c81d498f1a142398cb1693d7c3214aa9bdc84edc9315cc3c9c4e
Stable URL of this page
https://metraipolis.com/documento/dc_aebae90da23a

How to cite

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit — arXiv:2609.31631 — arXiv (en), https://arxiv.org/abs/2609.31631 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aebae90da23a (Metraipolis archive dc_aebae90da23a, sha256 cb4d6d4ec8e1c81d498f1a142398cb1693d7c3214aa9bdc84edc9315cc3c9c4e)
BibTeX
@misc{dcaebae90da23a,
  title = {OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit — arXiv:2609.31631},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31631},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aebae90da23a; sha256 cb4d6d4ec8e1c81d498f1a142398cb1693d7c3214aa9bdc84edc9315cc3c9c4e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31631. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33$\times$ faster search and 1.55$\times$ inference speedup. Codes will be available after acceptance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.31631 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.506Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).