Metraipolis › archive › dc_0ab8e8a7aeac
Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce — arXiv:2609.23898
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.23898
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T21:57:22.537Z
- SHA-256 of the archived text
a7fdb836d045c5e0a6c9310e90dea34db0cfba6beffa49efc5bdd6680c2b850f- Stable URL of this page
https://metraipolis.com/documento/dc_0ab8e8a7aeac
How to cite
Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce — arXiv:2609.23898 — arXiv (en), https://arxiv.org/abs/2609.23898 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ab8e8a7aeac (Metraipolis archive dc_0ab8e8a7aeac, sha256 a7fdb836d045c5e0a6c9310e90dea34db0cfba6beffa49efc5bdd6680c2b850f)
BibTeX
@misc{dc0ab8e8a7aeac,
title = {Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce — arXiv:2609.23898},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.23898},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ab8e8a7aeac; sha256 a7fdb836d045c5e0a6c9310e90dea34db0cfba6beffa49efc5bdd6680c2b850f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23898. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Ring All-Reduce is a widely used protocol for aggregating gradients across workers in large-scale distributed training. For a system with $N$ workers, its normalized per-worker communication is $2(N-1)/N$. In this work we present a communication-efficient Ring All-Reduce (CERAR) protocol for ``approximate'' gradient aggregation. CERAR partitions each local gradient into $c$ components and relies on linear encoding and decoding operations. It performs $L=c+N-2$ communication rounds over an $N$-worker ring, yielding normalized communication rate $1+(N-2)/c$ and normalized storage rate $1+N/c$. We present an explicit Vandermonde matrix based construction whose approximation error can be made arbitrarily small, with communication and storage rates approaching one as $c$ increases. However, this comes at the cost of ill-conditioned encoding matrices. We therefore give an alternate construction whose error is $O(\epsilon)$, while the relevant condition numbers are $O(\epsilon^{-r_\star})$, where $r_\star=\lceil c/N\rceil-1$. This motivates a condition-number-constrained optimization formulation for obtaining numerically stable practical designs. Our experiments show that, compared with the production-standard NCCL on NVIDIA A100 PCIe GPU clusters with low-bandwidth PCIe interconnect, CERAR reduces raw All-Reduce time by up to 47\%. For training a 1.4-billion-parameter Pythia model, it reduces training time by about 17\% for the same number of iterations while achieving essentially identical validation loss. On GPU clusters with high-bandwidth NVLink interconnects, CERAR performs worse in the All-Reduce phase and slightly worse in overall training time. However, the gap narrows for larger model sizes, suggesting that further gains may be possible with more optimized CERAR implementations.
Tema degli esploratori: matematica e informatica teorica (retro-catalogo arXiv: annunci del 2026-09-22 in math.IT).
Categorie arXiv: Information Theory (cs.IT).
Pagina del lavoro: https://arxiv.org/abs/2609.23898 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T21:44:07.475Z dalla pagina https://arxiv.org/catchup/math.IT/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).