Metraipolis › archive › dc_eb4ab9b862d6
The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch — arXiv:2610.38228
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.38228
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:18:58.020Z
- SHA-256 of the archived text
54c01a12139c188432b0fac304fcef29544de983cdd270814d08433b3459866d- Stable URL of this page
https://metraipolis.com/documento/dc_eb4ab9b862d6
How to cite
The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch — arXiv:2610.38228 — arXiv (en), https://arxiv.org/abs/2610.38228 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb4ab9b862d6 (Metraipolis archive dc_eb4ab9b862d6, sha256 54c01a12139c188432b0fac304fcef29544de983cdd270814d08433b3459866d)
BibTeX
@misc{dceb4ab9b862d6,
title = {The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch — arXiv:2610.38228},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.38228},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb4ab9b862d6; sha256 54c01a12139c188432b0fac304fcef29544de983cdd270814d08433b3459866d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38228. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG); Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2610.38228 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:53.367Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).