Metraipolis › archive › dc_748cd0f39b3a

Higher-order pruning of experts in mixture-of-experts language models — arXiv:2609.18916

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18916
License (as archived)
CC0-1.0
Archived
2026-10-15T15:04:56.215Z
SHA-256 of the archived text
2976207f217599b3a1555e10e2704450d8709fb8c5d0348378c6f541bb585db9
Stable URL of this page
https://metraipolis.com/documento/dc_748cd0f39b3a

How to cite

Higher-order pruning of experts in mixture-of-experts language models — arXiv:2609.18916 — arXiv (en), https://arxiv.org/abs/2609.18916 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_748cd0f39b3a (Metraipolis archive dc_748cd0f39b3a, sha256 2976207f217599b3a1555e10e2704450d8709fb8c5d0348378c6f541bb585db9)
BibTeX
@misc{dc748cd0f39b3a,
  title = {Higher-order pruning of experts in mixture-of-experts language models — arXiv:2609.18916},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18916},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_748cd0f39b3a; sha256 2976207f217599b3a1555e10e2704450d8709fb8c5d0348378c6f541bb585db9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18916. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective which provably minimizes an upper bound on the error resulting from pruning. We show that REAP (a state-of-the-art first-order pruning method) is a special case of HOPE where interaction terms are ignored. Across three frontier MoE models (up to 122B parameters), two distinct calibration sets, and multiple benchmarks (including math, instruction following, coding, and an agentic suite), we demonstrate that HOPE produces better pruning decisions than existing methods, and its advantage is most pronounced at high pruning rates and on challenging agentic workloads. At 50% pruning, HOPE outperforms all baselines and achieves an average rank of 1.58 out of 5 methods (versus 2.42 for the next-best method, REAP), with gains of up to +6.1% on agentic coding. Over all conditions, HOPE again achieves the best average rank and surpasses every other method in the majority of head-to-head comparisons. By preserving cooperative expert structure that first-order methods ignore, HOPE enables aggressive compression with minimal degradation, particularly on complex tasks where diverse expert combinations are invoked over long sequences.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.18916 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T14:08:19.157Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).