Metraipolis › archive › dc_09b78fa5eb7c

Towards a Statistical Understanding of Mixture-of-Experts — arXiv:2609.03501

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03501
License (as archived)
CC0-1.0
Archived
2026-10-17T02:08:39.396Z
SHA-256 of the archived text
3c4eb1df9cdd898221abeaf6c5a3e210286fedd042ca9264c532615e47bb2adb
Stable URL of this page
https://metraipolis.com/documento/dc_09b78fa5eb7c

How to cite

Towards a Statistical Understanding of Mixture-of-Experts — arXiv:2609.03501 — arXiv (en), https://arxiv.org/abs/2609.03501 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_09b78fa5eb7c (Metraipolis archive dc_09b78fa5eb7c, sha256 3c4eb1df9cdd898221abeaf6c5a3e210286fedd042ca9264c532615e47bb2adb)
BibTeX
@misc{dc09b78fa5eb7c,
  title = {Towards a Statistical Understanding of Mixture-of-Experts — arXiv:2609.03501},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03501},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_09b78fa5eb7c; sha256 3c4eb1df9cdd898221abeaf6c5a3e210286fedd042ca9264c532615e47bb2adb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03501. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.LG).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2609.03501 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T01:47:25.623Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).