Metraipolis › archive › dc_2a15e156281b
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts — arXiv:2609.09241
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.09241
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T07:25:04.601Z
- SHA-256 of the archived text
2c166ec7ae61423b95db4398fa6a070779f452d78f002c5cb6b7d79d3bc53a1e- Stable URL of this page
https://metraipolis.com/documento/dc_2a15e156281b
How to cite
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts — arXiv:2609.09241 — arXiv (en), https://arxiv.org/abs/2609.09241 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2a15e156281b (Metraipolis archive dc_2a15e156281b, sha256 2c166ec7ae61423b95db4398fa6a070779f452d78f002c5cb6b7d79d3bc53a1e)
BibTeX
@misc{dc2a15e156281b,
title = {Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts — arXiv:2609.09241},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.09241},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2a15e156281b; sha256 2c166ec7ae61423b95db4398fa6a070779f452d78f002c5cb6b7d79d3bc53a1e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09241. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top-$k$ expert selection policy, assigning the same expert budget to every token even when fewer experts may be sufficient. Inference-time dynamic top-$k$ routing can reduce computation without retraining, but existing methods often overlook the distributional shift caused by deviating from the training-time routing configuration. We show that reducing the number of activated experts consistently increases the RMS scale and variance of SMoE outputs, inducing a representation mismatch that contributes to downstream performance degradation in addition to the loss of expert capacity. To address this correctable component, we propose Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration. Across multiple SMoE LLMs, benchmarks, and routing strategies, LDA recovers much of the performance lost induced by the distributional shift under reduced routing while preserving sparse-inference efficiency with negligible overhead. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.09241 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T07:00:16.283Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).