Metraipolis › archive › dc_1b47cedbe6bc
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models? — arXiv:2609.33791
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33791
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:01:46.619Z
- SHA-256 of the archived text
5392e171d9582d2b25feb9a3c636308e0518111e3021262351b50b6de57ead44- Stable URL of this page
https://metraipolis.com/documento/dc_1b47cedbe6bc
How to cite
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models? — arXiv:2609.33791 — arXiv (en), https://arxiv.org/abs/2609.33791 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1b47cedbe6bc (Metraipolis archive dc_1b47cedbe6bc, sha256 5392e171d9582d2b25feb9a3c636308e0518111e3021262351b50b6de57ead44)
BibTeX
@misc{dc1b47cedbe6bc,
title = {Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models? — arXiv:2609.33791},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33791},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1b47cedbe6bc; sha256 5392e171d9582d2b25feb9a3c636308e0518111e3021262351b50b6de57ead44},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33791. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation method, OPD naturally inherits KL divergence as its standard loss. However, in this work, we find that KL divergence may not be necessary for OPD. We show that simply preserving the update direction is sufficient for effective OPD. As long as the update direction is toward the teacher, OPD works. More precisely, it is not the direction of every token, but the direction of a small subset of tokens where the teacher and student disagree strongly. We first show that simply assigning a reward of (+1) to tokens where the teacher probability is higher than the student probability and (-1) where it is lower, which merely encourages updates toward the teacher, reproduces almost the same training mode as OPD with reverse KL. We further show that only the direction of a small subset of tokens with large teacher-student disagreement is critical, and training works as long as their update direction is toward the teacher, even if other tokens are pulled away from the teacher. And as an application of these findings, we introduce Consensus Multi-Teacher On-Policy Distillation (C-MOPD) to improve Multi-Teacher On-Policy Distillation (MOPD). Unlike MOPD, which routes each sample to a single teacher and may cause capability conflicts across domains, C-MOPD lets every sample be supervised by all teachers. Experiments show that C-MOPD consistently outperforms MOPD on both math and code benchmarks. Our code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33791 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.747Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).