Metraipolis › archive › dc_2016b0d4d4b5
Unbiased Top-$k$ Estimation for On-Policy Distillation — arXiv:2609.34447
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34447
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:53:25.298Z
- SHA-256 of the archived text
adb7f6afdb513a12e679daa5a8516d749f502c51394b07ea97e263bef5ccdfcc- Stable URL of this page
https://metraipolis.com/documento/dc_2016b0d4d4b5
How to cite
Unbiased Top-$k$ Estimation for On-Policy Distillation — arXiv:2609.34447 — arXiv (en), https://arxiv.org/abs/2609.34447 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2016b0d4d4b5 (Metraipolis archive dc_2016b0d4d4b5, sha256 adb7f6afdb513a12e679daa5a8516d749f502c51394b07ea97e263bef5ccdfcc)
BibTeX
@misc{dc2016b0d4d4b5,
title = {Unbiased Top-$k$ Estimation for On-Policy Distillation — arXiv:2609.34447},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34447},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2016b0d4d4b5; sha256 adb7f6afdb513a12e679daa5a8516d749f502c51394b07ea97e263bef5ccdfcc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34447. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-$k$ OPD (TK-OPD) that use selected top-$k$ tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-$k$ tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-$k$ tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-$k$ On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-$k$ tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2609.34447 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.615Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).