Metraipolis › archive › dc_6b8aaf85583b

How Sparse Probability Maps Shape Mixture-of-Experts Routing — arXiv:2610.06677

Source
arXiv
Original
https://arxiv.org/abs/2610.06677
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:57.510Z
SHA-256 of the archived text
d7deb3b5beaf9abbcef359412d7900b923d3b0f7e522ebf846be284de048bc3d
Stable URL of this page
https://metraipolis.com/documento/dc_6b8aaf85583b

How to cite

How Sparse Probability Maps Shape Mixture-of-Experts Routing — arXiv:2610.06677 — arXiv, https://arxiv.org/abs/2610.06677 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6b8aaf85583b (Metraipolis archive dc_6b8aaf85583b, sha256 d7deb3b5beaf9abbcef359412d7900b923d3b0f7e522ebf846be284de048bc3d)
BibTeX
@misc{dc6b8aaf85583b,
  title = {How Sparse Probability Maps Shape Mixture-of-Experts Routing — arXiv:2610.06677},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06677},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6b8aaf85583b; sha256 d7deb3b5beaf9abbcef359412d7900b923d3b0f7e522ebf846be284de048bc3d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06677. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability maps such as sparsemax, alpha-entmax and normmax can adaptively assign exact zeros to selected experts, and therefore appear to offer token-dependent expert participation, even when using the same top-K machinery. In this work, we study whether and how this sparsity survives training. We train matched 300M and 1B top-2 MoE language models with softmax, 1.5-entmax, sparsemax and 2-normmax, and find that the maps behave very differently once trained: at 1B, entmax discards 30% less probability mass than softmax while almost never dropping a selected expert, sparsemax retains the most mass, and normmax routes 21% of tokens to a single expert. These outcomes are not properties of the maps alone. Each map drops a selected expert only when the gap between the two largest scores reaches a fixed threshold, and the trained routers differ in the score distribution they learn: the entmax router learns scores with roughly half the spread of softmax's, which keeps its top-2 gaps below its threshold, while sparsemax and normmax, which share the same threshold, learn different gap distributions and hence different participation. Routers thus co-adapt their scores to the map, and a map's capacity to produce zeros does not by itself determine expert participation. While none of the sparse maps improves validation loss over softmax, they make the trained models far less sensitive to selecting more experts at inference: sparsemax trained with K=2 loses 0.02 nats when run with K=8, where softmax loses 0.58. Our results indicate that adaptive MoE routing has to be designed around the joint behavior of the probability map and the learned scores, rather than around the map alone.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.06677 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.470Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).