Metraipolis › archive › dc_b6021fb797ad

Divergence controls entropy in distillation — arXiv:2610.03529

Source
arXiv
Original
https://arxiv.org/abs/2610.03529
License (as archived)
CC0-1.0
Archived
2026-10-05T22:52:02.082Z
SHA-256 of the archived text
c06f2956aaa86ed100f954e935a87139e36c9efdf76f6bf127a6e344dbb3ca09
Stable URL of this page
https://metraipolis.com/documento/dc_b6021fb797ad

How to cite

Divergence controls entropy in distillation — arXiv:2610.03529 — arXiv, https://arxiv.org/abs/2610.03529 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b6021fb797ad (Metraipolis archive dc_b6021fb797ad, sha256 c06f2956aaa86ed100f954e935a87139e36c9efdf76f6bf127a6e344dbb3ca09)
BibTeX
@misc{dcb6021fb797ad,
  title = {Divergence controls entropy in distillation — arXiv:2610.03529},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03529},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b6021fb797ad; sha256 c06f2956aaa86ed100f954e935a87139e36c9efdf76f6bf127a6e344dbb3ca09},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03529. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy of the student depends on the data and the divergence that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a special case, this yields an identity that we verify quantitatively in pretraining and supervised finetuning. Other divergences come with no such guarantee: reverse KL deflates entropy until the gap between student and teacher gets too large, and interpolating between the two changes entropy smoothly early in training but abruptly at convergence. The lower entropy of on-policy distillation comes from token-level reverse KL, not from on-policy sampling. The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.03529 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:46:10.632Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).