Metraipolis › archive › dc_ea4ae4680e48
Communication-Efficient LLM Adaptation over Decentralized GPU Meshes — arXiv:2609.14339
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14339
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T00:10:40.626Z
- SHA-256 of the archived text
a597cab638a4b0126a8ae70dd490b22531c890d76a27d488fa77313cc196ff46- Stable URL of this page
https://metraipolis.com/documento/dc_ea4ae4680e48
How to cite
Communication-Efficient LLM Adaptation over Decentralized GPU Meshes — arXiv:2609.14339 — arXiv (en), https://arxiv.org/abs/2609.14339 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea4ae4680e48 (Metraipolis archive dc_ea4ae4680e48, sha256 a597cab638a4b0126a8ae70dd490b22531c890d76a27d488fa77313cc196ff46)
BibTeX
@misc{dcea4ae4680e48,
title = {Communication-Efficient LLM Adaptation over Decentralized GPU Meshes — arXiv:2609.14339},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14339},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea4ae4680e48; sha256 a597cab638a4b0126a8ae70dd490b22531c890d76a27d488fa77313cc196ff46},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14339. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes the primary bottleneck. We study post-pretraining adaptation in this setting. We propose an asynchronous two-circuit system: a fast compressed training circuit drives throughput using activation masking for pipeline-parallel (PP) transfer and compressed data-parallel (DP) synchronization, while a slow anchor circuit runs occasional unmasked forward--backward passes off the critical path. Then, we introduce a spectral correction optimizer that uses these delayed anchor priors to denoise masked gradients without blocking the fast stream. Although prior work has found aggressive activation compression unreliable, we show that masking supports post-pretraining adaptation at high compression rates when anchored this way. Pipeline-parallel compression alone yields up to a $9\times$ throughput gain, and combining it with data-parallel compression increases beyond $40\times$ over internet-grade $\sim 200$Mbps connections, while matching dense uncompressed performance across domain adaptation and continual pretraining. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.14339 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T23:54:14.692Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).