Metraipolis › archive › dc_15e129011736

Hyperparameter Scaling Laws Across MoE Sparsity — arXiv:2609.08690

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.08690
License (as archived)
CC0-1.0
Archived
2026-10-16T12:58:16.595Z
SHA-256 of the archived text
60da0f80d59114ea47c981870697bc08e658ede75f732ab3ec4f964193bc81e6
Stable URL of this page
https://metraipolis.com/documento/dc_15e129011736

How to cite

Hyperparameter Scaling Laws Across MoE Sparsity — arXiv:2609.08690 — arXiv (en), https://arxiv.org/abs/2609.08690 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_15e129011736 (Metraipolis archive dc_15e129011736, sha256 60da0f80d59114ea47c981870697bc08e658ede75f732ab3ec4f964193bc81e6)
BibTeX
@misc{dc15e129011736,
  title = {Hyperparameter Scaling Laws Across MoE Sparsity — arXiv:2609.08690},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.08690},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_15e129011736; sha256 60da0f80d59114ea47c981870697bc08e658ede75f732ab3ec4f964193bc81e6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08690. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-embedding parameters, processing approximately 20 trillion tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, the optimal batch size follows a power-law relationship with training tokens $D$, whereas the optimal learning rate scales with training compute $C$ and remains robust to the allocation between model size and data. Across sparsity levels, the activation ratio $A$ enters both relationships as an additional multiplicative power-law factor. These observations lead to unified hyperparameter scaling laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from that of total expert count.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.08690 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.361Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).