Metraipolis › archive › dc_2a2aa4868ec6
Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02093
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:21:22.853Z
- SHA-256 of the archived text
50f5d6316c286bc360409fcf42565ac1745e6ff74aeddf91acfc439fd1bc936c- Stable URL of this page
https://metraipolis.com/documento/dc_2a2aa4868ec6
How to cite
Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093 — arXiv (en), https://arxiv.org/abs/2611.02093 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2a2aa4868ec6 (Metraipolis archive dc_2a2aa4868ec6, sha256 50f5d6316c286bc360409fcf42565ac1745e6ff74aeddf91acfc439fd1bc936c)
BibTeX
@misc{dc2a2aa4868ec6,
title = {Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02093},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2a2aa4868ec6; sha256 50f5d6316c286bc360409fcf42565ac1745e6ff74aeddf91acfc439fd1bc936c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02093. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak or uninformative teacher supervision. We introduce Bernoulli On-Policy Distillation (Bernoulli OPD), which directly matches the teacher and student probabilities for that token. This yields a bounded probability space correction that preserves teacher-student disagreement while avoiding the additional confidence-dependent suppression induced by vanilla OPD. Fixed-rollout analyses show that Bernoulli OPD substantially restores optimization signal to high-confidence tokens while remaining sensitive to teacher-student disagreement. We further analyze the resulting model behavior through confidence distributions, answer calibration, policy movement, and training dynamics, showing that Bernoulli OPD's gains cannot be explained simply by longer reasoning, larger optimization steps, or more uniform imitation of the teacher. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.02093 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:02:37.582Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).