Metraipolis › archive › dc_a6736ded0a5b

HyperParallel-FSDP: Topology-Aware Fully Sharded Training with Layout-Driven Muon on Ascend SuperPods — arXiv:2609.21594

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.21594
License (as archived): CC0-1.0
Archived: 2026-10-10T01:40:07.436Z
SHA-256 of the archived text: 9e59f6db818c0ae2d2a3a47f6cc9de858d735ed5a66f47d041430f1fc59d7fcb
Stable URL of this page: https://metraipolis.com/documento/dc_a6736ded0a5b

How to cite

HyperParallel-FSDP: Topology-Aware Fully Sharded Training with Layout-Driven Muon on Ascend SuperPods — arXiv:2609.21594 — arXiv (en), https://arxiv.org/abs/2609.21594 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a6736ded0a5b (Metraipolis archive dc_a6736ded0a5b, sha256 9e59f6db818c0ae2d2a3a47f6cc9de858d735ed5a66f47d041430f1fc59d7fcb)

BibTeX

@misc{dca6736ded0a5b,
  title = {HyperParallel-FSDP: Topology-Aware Fully Sharded Training with Layout-Driven Muon on Ascend SuperPods — arXiv:2609.21594},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21594},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a6736ded0a5b; sha256 9e59f6db818c0ae2d2a3a47f6cc9de858d735ed5a66f47d041430f1fc59d7fcb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21594. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Declarative SPMD programming uses tensor sharding descriptions to drive distributed execution, separating parallelization from model code. However, the evaluated PyTorch DTensor stack dispatches every operator below autograd, incurring repeated dispatch and metadata costs, while lacking an inexpensive end-to-end validation path. Existing FSDP and distributed Muon implementations also mismatch two-tier supernode topologies: FSDP relies on explicit parameter packing and unpacking, and Muon's whole-matrix orthogonalization conflicts with parameter sharding. We observe that distributed tensors need only express sharding semantics at the tensor API boundary above autograd, allowing differentiation and kernels to operate on plain tensors. Based on this insight, we present HyperParallel-FSDP, featuring: (1) dual-mode DTensor execution, using one sharding plan for both a production mode with one-time layout resolution and no steady-state dispatch overhead, and a validation mode with end-to-end metadata propagation, fail-fast checks, and gradient-equivalence testing; (2) topology-aware FSDP, with zero-copy intra-supernode collectives, fused inter-supernode reduction, and a cross-layer backward pipeline that avoids waits on slow links; and (3) layout-driven distributed Muon, with sharding-derived communication groups, deduplicated orthogonalization, and shape-fused Newton-Schulz iterations. On Atlas 900 A3 SuperPoD, HyperParallel-FSDP scales from 16 dies to 384 cards (768 ranks), sustaining 421k tokens/s for a 505B-parameter MoE while FSDP communication uses 2.9% of step time. It reduces mean step time by 29.7% versus PyTorch FSDP2 and 25.5% versus Megatron DDP, with Pearson correlation above 0.999997 over 1,000 steps. Distributed Muon improves profiler step time by 5.4-16.0% over competing systems. Source code is available at this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-21 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.21594 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:05:30.964Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).