Metraipolis › archive › dc_cd66fa1ce86e

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training — arXiv:2611.11238

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.11238
License (as archived): CC0-1.0
Archived: 2026-11-10T18:43:13.725Z
SHA-256 of the archived text: 34c9d2a0c44fc38b5322826abfdd2b8dd056cb8b9e55a38aa4e5dff775e1da50
Stable URL of this page: https://metraipolis.com/documento/dc_cd66fa1ce86e

How to cite

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training — arXiv:2611.11238 — arXiv (en), https://arxiv.org/abs/2611.11238 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd66fa1ce86e (Metraipolis archive dc_cd66fa1ce86e, sha256 34c9d2a0c44fc38b5322826abfdd2b8dd056cb8b9e55a38aa4e5dff775e1da50)

BibTeX

@misc{dccd66fa1ce86e,
  title = {FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training — arXiv:2611.11238},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11238},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd66fa1ce86e; sha256 34c9d2a0c44fc38b5322826abfdd2b8dd056cb8b9e55a38aa4e5dff775e1da50},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11238. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Asynchronous pipeline parallelism can accelerate distributed model training, but suffers from instability caused by inconsistent gradient computations. We identify a fundamental source of this inconsistency: error signals are coupled to forward-time features, not just parameter states. This means that parameters may evolve as long as they preserve forward-equivalence. Leveraging this insight, we introduce FRESCO, which frames asynchronous updates as a constrained optimization problem: it finds the minimal update modification that acts trivially on the activation subspace. We demonstrate that FRESCO consistently outperforms representative asynchronous baselines in LLM pre-training, with the gains most pronounced under deep pipelines and high inconsistency levels: at pipeline depth 32, it reaches the same loss as the strongest baseline with 43% fewer iterations and 30% less wall-clock time, while requiring 8~921$\times$ less memory overhead. Moving beyond the current limitations, FRESCO establishes a new foundation for high-utilization pipeline parallel training, providing a scalable path for large-model development on diverse, resource-constrained systems.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.11238 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.631Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).