Metraipolis › archive › dc_e1604df689a2

What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning — arXiv:2610.41366

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41366
License (as archived): CC0-1.0
Archived: 2026-11-03T07:35:13.965Z
SHA-256 of the archived text: a7675a5bb91f838b21b0a159fb779711442d7ba43336ae2886d138244fc8c8b7
Stable URL of this page: https://metraipolis.com/documento/dc_e1604df689a2

How to cite

What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning — arXiv:2610.41366 — arXiv (en), https://arxiv.org/abs/2610.41366 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e1604df689a2 (Metraipolis archive dc_e1604df689a2, sha256 a7675a5bb91f838b21b0a159fb779711442d7ba43336ae2886d138244fc8c8b7)

BibTeX

@misc{dce1604df689a2,
  title = {What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning — arXiv:2610.41366},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41366},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e1604df689a2; sha256 a7675a5bb91f838b21b0a159fb779711442d7ba43336ae2886d138244fc8c8b7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41366. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.41366 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:29:48.000Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).