Metraipolis › archive › dc_5c368deb7fbc

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network — arXiv:2609.01952

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01952
License (as archived)
CC0-1.0
Archived
2026-10-17T03:26:59.116Z
SHA-256 of the archived text
7e4f2a09f900c0a9a7d50ba38d17abf94e1b4fca7402ee0bcda2ec5775ea3f05
Stable URL of this page
https://metraipolis.com/documento/dc_5c368deb7fbc

How to cite

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network — arXiv:2609.01952 — arXiv (en), https://arxiv.org/abs/2609.01952 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5c368deb7fbc (Metraipolis archive dc_5c368deb7fbc, sha256 7e4f2a09f900c0a9a7d50ba38d17abf94e1b4fca7402ee0bcda2ec5775ea3f05)
BibTeX
@misc{dc5c368deb7fbc,
  title = {Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network — arXiv:2609.01952},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01952},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5c368deb7fbc; sha256 7e4f2a09f900c0a9a7d50ba38d17abf94e1b4fca7402ee0bcda2ec5775ea3f05},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01952. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate $O(1/(\eta T))$ to an $O(\eta)+O(B_f^2)+O(\zeta_f^2)$ neighbourhood. Here $B_f$ is the distance from the task optimum to the peers' reachable classes and $\zeta_f$ measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by $40-61\times$, while isolated training does not. The sampled stationarity diagnostic has late transient exponents $0.99-1.90$ on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.01952 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T03:13:12.492Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).