Metraipolis › archive › dc_fcc02418dace

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts — arXiv:2608.10605

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.10605
License (as archived): CC0-1.0
Archived: 2026-10-19T06:09:20.566Z
SHA-256 of the archived text: 780dc87937a8e1972a22028bd88a64b1e5c9d789168b9f1b4c0cffd586d0e97a
Stable URL of this page: https://metraipolis.com/documento/dc_fcc02418dace

How to cite

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts — arXiv:2608.10605 — arXiv (en), https://arxiv.org/abs/2608.10605 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcc02418dace (Metraipolis archive dc_fcc02418dace, sha256 780dc87937a8e1972a22028bd88a64b1e5c9d789168b9f1b4c0cffd586d0e97a)

BibTeX

@misc{dcfcc02418dace,
  title = {Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts — arXiv:2608.10605},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10605},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcc02418dace; sha256 780dc87937a8e1972a22028bd88a64b1e5c9d789168b9f1b4c0cffd586d0e97a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10605. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency. In this work, we develop MOSAIC, which formulates model architecture and systems co-design as an optimization problem. MOSAIC couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout. We instantiate the framework for sparse Mixture-of-Experts (MoE) language models, where expert count, routing sparsity, and other MoE layer dimensions affect both the loss and systems efficiency. We fit a scaling law on sparse MoE models trained on text data, whose scaling dimensions include the sparsity factor, which is the fraction of model parameters inactive per token in a forward pass. The scaling law sweeps in our work span active parameters from $104$ million to $2.7$ billion and total model sizes reaching $79$ billion parameters. We show that, within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support. An optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. Our results argue for a shift towards unified architecture and systems co-design for frontier language model training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.10605 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.389Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).