Metraipolis › archive › dc_a3cb6153a7ee
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models — arXiv:2610.27123
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.27123
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-21T07:07:28.463Z
- SHA-256 of the archived text
3559272633edbedbeb8e213b374e64d0e6d0e92a69eebed39aa4661c5568330c- Stable URL of this page
https://metraipolis.com/documento/dc_a3cb6153a7ee
How to cite
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models — arXiv:2610.27123 — arXiv (en), https://arxiv.org/abs/2610.27123 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a3cb6153a7ee (Metraipolis archive dc_a3cb6153a7ee, sha256 3559272633edbedbeb8e213b374e64d0e6d0e92a69eebed39aa4661c5568330c)
BibTeX
@misc{dca3cb6153a7ee,
title = {Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models — arXiv:2610.27123},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.27123},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a3cb6153a7ee; sha256 3559272633edbedbeb8e213b374e64d0e6d0e92a69eebed39aa4661c5568330c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27123. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.27123 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-21T07:05:43.525Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).