Metraipolis › archive › dc_b3a02e017d28
Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs — arXiv:2610.28369
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.28369
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-21T07:08:27.217Z
- SHA-256 of the archived text
bb88df2cfcb1c0f31bd12d5bdb14c79eee6d7a8021b9fa90ab0d7f19ddf2e4b9- Stable URL of this page
https://metraipolis.com/documento/dc_b3a02e017d28
How to cite
Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs — arXiv:2610.28369 — arXiv (en), https://arxiv.org/abs/2610.28369 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b3a02e017d28 (Metraipolis archive dc_b3a02e017d28, sha256 bb88df2cfcb1c0f31bd12d5bdb14c79eee6d7a8021b9fa90ab0d7f19ddf2e4b9)
BibTeX
@misc{dcb3a02e017d28,
title = {Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs — arXiv:2610.28369},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.28369},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b3a02e017d28; sha256 bb88df2cfcb1c0f31bd12d5bdb14c79eee6d7a8021b9fa90ab0d7f19ddf2e4b9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.28369. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per token during inference, aiming to preserve model quality while further reducing the number of active parameters. These methods have added calibrated importance scores, inter-expert similarity metrics, or learned per-token allocators in order to identify experts to skip. However, different calibration regimes, different reductions in the effective number of experts, and evaluation suites make comparisons difficult. We classify four representative prior methods into a unified taxonomy and benchmark them against a natural, simple baseline: statically reducing the number of experts $k$ (i.e., same reduced $k$ across all layers and tokens). We provide a fair comparison of these methods at matched expert-compute budget, on two modern MoEs (Gemma4-26B-A4B and Qwen3-30B-A3B) in a text-only setting and a contemporary vision-language model (Qwen3-VL) in a multimodal setting. Surprisingly, we find that the simple baseline is hard to outperform consistently across models, modalities, benchmarks, and calibration settings. The methods we benchmark often underperform the baseline and achieved gains, when present, are minor. We further find that the effective $k$ achieved (or optimized for) in calibration can diverge from the effective $k$ in testing, resulting in over- or under-estimating the compute at inference time. Based on these findings, we recommend a calibration-free per-modality static $k$ reduction and validate this on a held-out model and modality. We further argue that a static control at matched measured compute is the baseline any dynamic skipping method must beat. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.28369 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-21T07:05:43.646Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).