Metraipolis › archive › dc_03391972cbfd

SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs — arXiv:2609.32197

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.32197
License (as archived): CC0-1.0
Archived: 2026-10-07T22:12:01.203Z
SHA-256 of the archived text: a6d6e56079d88e3460e38bc0a2294e4682eb8d96a3703fbc594f1d54593a8a4b
Stable URL of this page: https://metraipolis.com/documento/dc_03391972cbfd

How to cite

SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs — arXiv:2609.32197 — arXiv (en), https://arxiv.org/abs/2609.32197 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_03391972cbfd (Metraipolis archive dc_03391972cbfd, sha256 a6d6e56079d88e3460e38bc0a2294e4682eb8d96a3703fbc594f1d54593a8a4b)

BibTeX

@misc{dc03391972cbfd,
  title = {SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs — arXiv:2609.32197},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32197},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_03391972cbfd; sha256 a6d6e56079d88e3460e38bc0a2294e4682eb8d96a3703fbc594f1d54593a8a4b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32197. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion Transformers have become the dominant architecture for video generation. Their substantial computational cost motivates scaling inference across multi-GPU servers, yet efficient scaling remains challenging on commodity GPUs connected via PCIe, whose bandwidth is limited. Although sparse attention substantially reduces computation, its implications for communication remain underexplored. This paper argues that attention sparsity should be treated as a communication primitive. We present SparSP, an efficient sparse sequence parallel communication system that co-designs token distribution, communication routing, and asynchronous execution for sparse video diffusion models. First, Dependency-Aware Placement distributes sequence blocks according to diffusion models' sparse attention patterns. Second, Demand-Directed KV Routing transfers KV blocks directly to requesting GPUs without intermediate relays. Third, a Decoupled Transfer Runtime separates communication from GPU computation to reduce resource contention and maximize effective bandwidth. Our evaluation shows that SparSP improves attention performance by 1.38- 1.5$\times$, achieves an average 1.17$\times$ (up to 1.69$\times$) end-to-end speedup across three representative servers and three video diffusion models, and reduces communication volume by 12.54-23.05%. Moreover, we achieve an average 1.53-1.76$\times$ bandwidth improvement over NCCL.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-29 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.32197 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:53:13.642Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).