Metraipolis › archive › dc_783dcfa89c5e
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts — arXiv:2610.11575
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11575
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:08:30.947Z
- SHA-256 of the archived text
5fe8d0100148802553b72780cdb059586c11e33bbfaafd4b56ba7269a49edbb4- Stable URL of this page
https://metraipolis.com/documento/dc_783dcfa89c5e
How to cite
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts — arXiv:2610.11575 — arXiv (en), https://arxiv.org/abs/2610.11575 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_783dcfa89c5e (Metraipolis archive dc_783dcfa89c5e, sha256 5fe8d0100148802553b72780cdb059586c11e33bbfaafd4b56ba7269a49edbb4)
BibTeX
@misc{dc783dcfa89c5e,
title = {Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts — arXiv:2610.11575},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11575},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_783dcfa89c5e; sha256 5fe8d0100148802553b72780cdb059586c11e33bbfaafd4b56ba7269a49edbb4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11575. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sparse Mixture-of-Experts models scale parameter capacity efficiently while maintaining a fixed compute budget per token. However, traditional training paradigms enforce a static choice of top-$k$ experts, which converts a continuous routing distribution into a rigid step function. This constraint introduces a brittle boundary where highly competitive experts are arbitrarily separated into full-supervision and zero-feedback zones based on minor score fluctuations. To address this issue, we propose Elastic Expert Routing, which stochastically samples the active expert budget from a localized discrete distribution centered at $k$. Over multiple training iterations, this mechanism softens the sharp threshold into a gradual probability distribution. Because the sampling neighborhood remains symmetric, this approach matches the expected computational cost of deterministic training, while preserving the inference budget. Extensive experiments demonstrate the efficacy of our method on both supervised fine-tuning and from-scratch pretraining settings. During supervised fine-tuning, elastic routing improves downstream macro-averages on OLMoE-1B-7B and Qwen3-30B-A3B by $+0.84$ and $+2.02$ points, respectively. In addition, in from-scratch pretraining, it outperforms the static top-$k$ baseline by $1.6$ points on average across downstream tasks. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.11575 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:15.219Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).