Metraipolis › archive › dc_a5d643a186ad
Rethinking Reverse KL as Adaptive Entropy Distillation — arXiv:2608.14685
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14685
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T19:39:49.654Z
- SHA-256 of the archived text
95b508fd7b92b0bccd361d78a11ee9f0d5cf891e392c42b57208fb789ab5bbfd- Stable URL of this page
https://metraipolis.com/documento/dc_a5d643a186ad
How to cite
Rethinking Reverse KL as Adaptive Entropy Distillation — arXiv:2608.14685 — arXiv (en), https://arxiv.org/abs/2608.14685 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a5d643a186ad (Metraipolis archive dc_a5d643a186ad, sha256 95b508fd7b92b0bccd361d78a11ee9f0d5cf891e392c42b57208fb789ab5bbfd)
BibTeX
@misc{dca5d643a186ad,
title = {Rethinking Reverse KL as Adaptive Entropy Distillation — arXiv:2608.14685},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14685},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a5d643a186ad; sha256 95b508fd7b92b0bccd361d78a11ee9f0d5cf891e392c42b57208fb789ab5bbfd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14685. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In particular, existing methods mainly combine FKL and RKL, overlooking that RKL itself provides a mechanism for adjusting the student's imitation strength. Motivated by this, we revisit on-policy Reverse Kullback-Leibler (RKL) distillation and decompose its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch. We show theoretically that the token-level optimal student distribution corresponds to a tempered variant of the teacher distribution, where the adaptive weight controls the trade-off between mode-seeking and uncertainty preservation. Guided by this insight, we propose \textbf{Adaptive Entropy Distillation (AED)}, which uses the teacher's entropy to dynamically calibrate token-level imitation strength. Experiments on instruction-following and mathematical reasoning benchmarks demonstrate that AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.14685 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T19:02:44.042Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).