Metraipolis › archive › dc_bf91c5c11f63
HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale — arXiv:2609.34318
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34318
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:12:11.576Z
- SHA-256 of the archived text
279aacbf47702a12eac68af6de08c9818f679e481a201a2f0f6a61e20d5fe97a- Stable URL of this page
https://metraipolis.com/documento/dc_bf91c5c11f63
How to cite
HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale — arXiv:2609.34318 — arXiv (en), https://arxiv.org/abs/2609.34318 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf91c5c11f63 (Metraipolis archive dc_bf91c5c11f63, sha256 279aacbf47702a12eac68af6de08c9818f679e481a201a2f0f6a61e20d5fe97a)
BibTeX
@misc{dcbf91c5c11f63,
title = {HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale — arXiv:2609.34318},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34318},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf91c5c11f63; sha256 279aacbf47702a12eac68af6de08c9818f679e481a201a2f0f6a61e20d5fe97a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34318. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context training runs on sequences whose lengths span orders of magnitude, and dynamic context parallelism (DCP) gives each sequence its own CP degree. Existing DCP systems either do not scale or perform poorly on mainstream models, leaving Megatron-Core (Mcore) DCP as the only option at production scale. Mcore DCP, however, sizes each degree to fit memory, which grows linearly with length while attention grows quadratically, so comparable token counts hide unequal computation. In our production 256K-context training job on more than 11K GPUs under Mcore DCP, per-rank microbatch times differ by up to 5x at similar token counts. The skew leads to a 46% pipeline bubble and a 13% data-parallel bubble. We present HyDra, a scalable load-driven DCP system. Its scheduler balances computation by pulling every rank toward one load target, and balance in turn makes that target solvable in closed form. It places sequences with lazy heaps rather than whole-pool scans. That balance asks for CP degrees larger than memory requires, so its CP engine nests an inner Ulysses group in a shallow outer ring, letting a higher degree lower computation and communication together. Evaluation at both scales shows consistent gains. On a 512-GPU testbed, HyDra raises throughput over Mcore DCP by 1.18x on average at 32K context and 2.48x at 256K. On a 2,048-GPU production job, it shrinks the pipeline bubble from 36% to 14%, cuts scheduling time by 2.3x, and raises throughput by 1.10-1.43x (avg. 1.25x) over Mcore DCP and 1.33-1.90x (avg. 1.59x) over static CP. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-29 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.34318 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:53:13.705Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).