Metraipolis › archive › dc_ad381df64725
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts — arXiv:2608.20061
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.20061
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T11:39:33.070Z
- SHA-256 of the archived text
cedec3e866ec7c4a757a31f042e688d5043b1c4b580faa4abf7038272b06f54f- Stable URL of this page
https://metraipolis.com/documento/dc_ad381df64725
How to cite
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts — arXiv:2608.20061 — arXiv (en), https://arxiv.org/abs/2608.20061 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ad381df64725 (Metraipolis archive dc_ad381df64725, sha256 cedec3e866ec7c4a757a31f042e688d5043b1c4b580faa4abf7038272b06f54f)
BibTeX
@misc{dcad381df64725,
title = {Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts — arXiv:2608.20061},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.20061},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ad381df64725; sha256 cedec3e866ec7c4a757a31f042e688d5043b1c4b580faa4abf7038272b06f54f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20061. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost. However, optimizing their hyperparameters---particularly the learning rate---at extreme scales of both model size and token budget via sweeping remains computationally prohibitive. In this paper, we propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models by transferring them across scaling model widths, and subsequently extrapolating to trillion-token horizons. First, we formulate a Maximal Update Parameterization ($\mu$P) adaptation for MoE architectures utilizing Multi-head Latent Attention (MLA) and the Muon optimizer, demonstrating that optimal learning rates transfer consistently across width-scaled models. Second, we extend this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to the optimal values derived from small proxy models on limited budgets, we successfully extrapolate the ideal learning rate to massive training horizons (e.g., 10 trillion tokens) with high fidelity ($R^2=0.95$). Consequently, this indicates that proxy training on small models is sufficient to determine the optimal learning rate for the extensive training of large-scale MoEs. We apply the proposed methodology to pretrain our foundation model (155B total, 17B active parameters) from scratch, and the stable training and evaluation results validate that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.20061 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T11:38:27.958Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).