Metraipolis › archive › dc_4ecc832eaaab

When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts — arXiv:2611.01602

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01602
License (as archived): CC0-1.0
Archived: 2026-11-03T11:25:06.706Z
SHA-256 of the archived text: 4c62858f516e9bfe143befe8cde4848fe5c13dce91222f870b08216ea9e1d70c
Stable URL of this page: https://metraipolis.com/documento/dc_4ecc832eaaab

How to cite

When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts — arXiv:2611.01602 — arXiv (en), https://arxiv.org/abs/2611.01602 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4ecc832eaaab (Metraipolis archive dc_4ecc832eaaab, sha256 4c62858f516e9bfe143befe8cde4848fe5c13dce91222f870b08216ea9e1d70c)

BibTeX

@misc{dc4ecc832eaaab,
  title = {When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts — arXiv:2611.01602},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01602},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4ecc832eaaab; sha256 4c62858f516e9bfe143befe8cde4848fe5c13dce91222f870b08216ea9e1d70c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01602. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sparse mixtures of experts combine multiple prediction models while evaluating only a small subset for each input. Although this architecture reduces computation, how expert separation and routing sparsity jointly govern expert recovery remains an open problem. In this paper, we address this question for location Gaussian mixtures of experts with a fixed number of experts by establishing how expert separation and spatial arrangement determine recovery rates. For dense models, our results reveal that when experts form well-separated clusters, the separation geometry is governed by the size of the largest cluster; without a prescribed cluster structure, this geometry is instead driven by the total number of experts. For sparse models, the separation geometry on retained routing regions is determined by the number of active experts, while the contribution of omitted regions is controlled by their total probability. Consequently, activating fewer experts reduces the separation complexity in these bounds. Furthermore, Top-1 routing yields a particularly striking result: experts can be recovered at a rate independent of the separation quantity. Our technical approach combines polynomial interpolation and Gaussian reproducing-kernel Hilbert space (RKHS) test functions with quantitative restriction inequalities that extend pointwise comparisons to expert functions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01602 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.821Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).