Metraipolis › archive › dc_b7792ca25fa7
The Distributional View of Knowledge Distillation — arXiv:2608.15215
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.15215
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T19:44:06.925Z
- SHA-256 of the archived text
680f8527d3e1db763fcf25db4818ba2ce3ddc1b682c5f24bcf87adb535fc9143- Stable URL of this page
https://metraipolis.com/documento/dc_b7792ca25fa7
How to cite
The Distributional View of Knowledge Distillation — arXiv:2608.15215 — arXiv (en), https://arxiv.org/abs/2608.15215 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b7792ca25fa7 (Metraipolis archive dc_b7792ca25fa7, sha256 680f8527d3e1db763fcf25db4818ba2ce3ddc1b682c5f24bcf87adb535fc9143)
BibTeX
@misc{dcb7792ca25fa7,
title = {The Distributional View of Knowledge Distillation — arXiv:2608.15215},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.15215},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b7792ca25fa7; sha256 680f8527d3e1db763fcf25db4818ba2ce3ddc1b682c5f24bcf87adb535fc9143},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15215. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass. We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views - marginals of the annealing path of its logits - and the student is trained against a geometry-aware aggregate of these views under an embedding-based ground cost. We formalize the resulting design space (mixtures, log-linear pooling, entropic Wasserstein barycenters, and a debiased Sinkhorn-divergence flagship in hub and path forms), prove an exact collapse result showing log-linear pooling of tempered views is equivalent to a single temperature, and give a multi-marginal Schrodinger-bridge reading that yields falsifiable predictions. On instruction-tuned Pythia pairs, experiments yield three empirical laws: (i) dispersion law - the benefit of multi-temperature aggregation grows monotonically with the effective temperature dispersion of the views, not with their number; (ii) dispersed views unlock the aggregation operator - the barycenter separates from the arithmetic mixture exactly when transport-based aggregation starts to beat averaging; and (iii) two-regime picture governed by the ceiling gap $\Gamma=\mathrm{PPL}_{\mathrm{SFT}}-\mathrm{PPL}_{T}$: when the fine-tuned teacher barely beats a supervised student the gentle transport objective is the best KD loss but no KD beats supervised fine-tuning, whereas at a real ceiling the ranking inverts - and the sign of the fidelity-generalization correlation flips. We argue that "which distillation loss is the best" is not a fixed property of the loss but a function of $\Gamma$.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.LG).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.15215 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T19:02:44.254Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).