Metraipolis › archive › dc_7178cae693ee

Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery — arXiv:2610.39442

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.39442
License (as archived)
CC0-1.0
Archived
2026-11-01T02:25:05.966Z
SHA-256 of the archived text
7bd673bb85fa841616e5ee4fad903aa4153ce35d8a769cf3bb49ae9613e526dc
Stable URL of this page
https://metraipolis.com/documento/dc_7178cae693ee

How to cite

Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery — arXiv:2610.39442 — arXiv (en), https://arxiv.org/abs/2610.39442 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7178cae693ee (Metraipolis archive dc_7178cae693ee, sha256 7bd673bb85fa841616e5ee4fad903aa4153ce35d8a769cf3bb49ae9613e526dc)
BibTeX
@misc{dc7178cae693ee,
  title = {Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery — arXiv:2610.39442},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39442},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7178cae693ee; sha256 7bd673bb85fa841616e5ee4fad903aa4153ce35d8a769cf3bb49ae9613e526dc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39442. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.39442 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.166Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).