Metraipolis › archive › dc_8df08daf374f
CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training — arXiv:2609.04609
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04609
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:41:12.861Z
- SHA-256 of the archived text
a21031feea962793a5e94d745bee3d6a3368709a9f2035d4672636770e504377- Stable URL of this page
https://metraipolis.com/documento/dc_8df08daf374f
How to cite
CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training — arXiv:2609.04609 — arXiv (en), https://arxiv.org/abs/2609.04609 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8df08daf374f (Metraipolis archive dc_8df08daf374f, sha256 a21031feea962793a5e94d745bee3d6a3368709a9f2035d4672636770e504377)
BibTeX
@misc{dc8df08daf374f,
title = {CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training — arXiv:2609.04609},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04609},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8df08daf374f; sha256 a21031feea962793a5e94d745bee3d6a3368709a9f2035d4672636770e504377},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04609. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In training Mixture-of-Experts (MoE) models, sharded data parallelism partitions each expert's parameters across GPUs, requiring an Allgather operation to reconstruct the full weight matrix before each layer executes. This communication often dominates iteration time. Prior work often reduces this overhead using lossy compression methods that sacrifice numerical fidelity, while existing lossless methods do not exploit cross-iteration exponent stability. In this paper, we propose Cross-Iteration Exponent Reuse Allgather (CIERA), a lossless, system-aware communication method for sharded MoE training. We observe that after a brief warmup phase, the exponent values of most weights remain unchanged across iterations. Based on this, we cache exponents locally and transmit only the sign and mantissa when exponents are unchanged. The receiver reconstructs the original weights exactly by combining the cached exponents with the received data. Since parameter matrices vary in shape across layers, compression is applied only when it yields net time savings. Moreover, the compression operation is overlapped with both Allgather communication and computation. Our real experiments and large-scale trace-driven simulator show that on OLMoE-1B-7B at 16 GPUs, CIERA achieves a 3.70x speedup over the lossless baseline and 3.68x over the lossy baseline, projected to reach 4.28x and 4.42x respectively at 128 GPUs, while preserving bitwise-exact parameter reconstruction in all evaluated runs. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-07 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.04609 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:26:34.114Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).