Metraipolis › archive › dc_529fb202b80b

Accelerating Sharded Data Parallelism at Scale with Federated Learning — arXiv:2609.20359

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.20359
License (as archived)
CC0-1.0
Archived
2026-10-10T03:24:15.585Z
SHA-256 of the archived text
6d3d8cc15a6acf8fda9455b54f1bdc8dcf40cdfc7c2ab9e2d7675c18316eda61
Stable URL of this page
https://metraipolis.com/documento/dc_529fb202b80b

How to cite

Accelerating Sharded Data Parallelism at Scale with Federated Learning — arXiv:2609.20359 — arXiv (en), https://arxiv.org/abs/2609.20359 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_529fb202b80b (Metraipolis archive dc_529fb202b80b, sha256 6d3d8cc15a6acf8fda9455b54f1bdc8dcf40cdfc7c2ab9e2d7675c18316eda61)
BibTeX
@misc{dc529fb202b80b,
  title = {Accelerating Sharded Data Parallelism at Scale with Federated Learning — arXiv:2609.20359},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.20359},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_529fb202b80b; sha256 6d3d8cc15a6acf8fda9455b54f1bdc8dcf40cdfc7c2ab9e2d7675c18316eda61},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.20359. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups' size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.20359 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:04:13.946Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).