Metraipolis › archive › dc_11cd3ddde5fa

AutoParallel: Composable, PyTorch-Native Automatic Parallelism for Production Training — arXiv:2611.08660

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.08660
License (as archived): CC0-1.0
Archived: 2026-11-09T22:36:27.709Z
SHA-256 of the archived text: 0e9b816afb7dc2edaef9d633534bcf5ebf3f0fba30745456c238fb0ae28b19f6
Stable URL of this page: https://metraipolis.com/documento/dc_11cd3ddde5fa

How to cite

AutoParallel: Composable, PyTorch-Native Automatic Parallelism for Production Training — arXiv:2611.08660 — arXiv (en), https://arxiv.org/abs/2611.08660 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_11cd3ddde5fa (Metraipolis archive dc_11cd3ddde5fa, sha256 0e9b816afb7dc2edaef9d633534bcf5ebf3f0fba30745456c238fb0ae28b19f6)

BibTeX

@misc{dc11cd3ddde5fa,
  title = {AutoParallel: Composable, PyTorch-Native Automatic Parallelism for Production Training — arXiv:2611.08660},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08660},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_11cd3ddde5fa; sha256 0e9b816afb7dc2edaef9d633534bcf5ebf3f0fba30745456c238fb0ae28b19f6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08660. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training large models requires multiple interacting forms of parallelism. As models and clusters scale, finding and maintaining efficient parallelization plans becomes essential for training performance. Diverse model architectures, workloads, and hardware topologies demand flexible strategies, yet production PyTorch systems typically rely on hand-written recipes that are costly to adapt and tend to reuse familiar patterns, which may be suboptimal for a given workload. We present AutoParallel, a PyTorch-native planner for distributed model training. AutoParallel searches parallelization choices jointly across the training computation, allowing established techniques to be combined and unconventional strategies to emerge. It expresses its decisions using native PyTorch abstractions and produces ordinary PyTorch programs, avoiding a separate execution runtime while composing directly with the PyTorch compilation and training stack. AutoParallel also composes automatic decisions with model-specific distributed code, allowing dynamic or specialized regions to remain under user control. Plans are inspectable and replayable artifacts. Across dense, hybrid, and mixture-of-experts models on meshes of up to four dimensions and 128 NVIDIA H100 GPUs, AutoParallel matches or exceeds expert-written TorchTitan recipes on most configurations, with throughput gains of up to 18.3%, and discovers plans that uniform expert recipes do not express.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.08660 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:31:17.028Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).