Metraipolis › archive › dc_5c504ccd53ec
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training — arXiv:2609.14306
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14306
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T23:17:36.416Z
- SHA-256 of the archived text
1dbe42dcd7ee22dfa9d3b4e86a8d40faee2b7479372ac7019cc5c15ccdea9afd- Stable URL of this page
https://metraipolis.com/documento/dc_5c504ccd53ec
How to cite
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training — arXiv:2609.14306 — arXiv (en), https://arxiv.org/abs/2609.14306 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5c504ccd53ec (Metraipolis archive dc_5c504ccd53ec, sha256 1dbe42dcd7ee22dfa9d3b4e86a8d40faee2b7479372ac7019cc5c15ccdea9afd)
BibTeX
@misc{dc5c504ccd53ec,
title = {Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training — arXiv:2609.14306},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14306},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5c504ccd53ec; sha256 1dbe42dcd7ee22dfa9d3b4e86a8d40faee2b7479372ac7019cc5c15ccdea9afd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14306. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint. Four are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count. Which one runs out first changes with the model, the context length, and the device count, so lowering the largest only exposes the next. We bound all four with schedules whose GPU working set is fixed at launch: PipelinedLLEP extends least-loaded expert parallelism with a cap on the tokens each source contributes to a dispatch chunk, Ring-DTP circulates activations or weight shards around a ring at the vocabulary projection and folds each block of logits into an online log-sum-exp, Selective checkpoint offload (SCO) keeps the one long-lived tensor of each checkpoint boundary in CPU memory, and OffloadStreamAdamW turns the serial CPU Adam update of optimizer offload into a bucket pipeline. All four change only the order and granularity of computation and data movement, so the loss and gradients stay exact. In matched component tests, they cut the MoE dispatch peak by up to $59.3\%$ without losing throughput, the vocabulary projection peak by $86.6\%$, and the offloaded optimizer step by $2.05\times$ faster. Composed on MoE models from 120B to 667B parameters, they train at 1M context length, $8$--$32\times$ the reach of a tuned FSDP2 baseline, and up to $10.4\times$ its throughput. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-15 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.14306 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:54:12.970Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).