Metraipolis › archive › dc_a22ec8cf0ee9

Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02093
License (as archived)
CC0-1.0
Archived
2026-11-03T18:37:10.407Z
SHA-256 of the archived text
a44df133dcdbef234d6e418c5b7ca996b22755deae79ccc70074c7b17948181d
Stable URL of this page
https://metraipolis.com/documento/dc_a22ec8cf0ee9

How to cite

Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093 — arXiv (en), https://arxiv.org/abs/2611.02093 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a22ec8cf0ee9 (Metraipolis archive dc_a22ec8cf0ee9, sha256 a44df133dcdbef234d6e418c5b7ca996b22755deae79ccc70074c7b17948181d)
BibTeX
@misc{dca22ec8cf0ee9,
  title = {Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens — arXiv:2611.02093},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02093},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a22ec8cf0ee9; sha256 a44df133dcdbef234d6e418c5b7ca996b22755deae79ccc70074c7b17948181d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02093. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak or uninformative teacher supervision. We introduce Bernoulli On-Policy Distillation (Bernoulli OPD), which directly matches the teacher and student probabilities for that token. This yields a bounded probability space correction that preserves teacher-student disagreement while avoiding the additional confidence-dependent suppression induced by vanilla OPD. Fixed-rollout analyses show that Bernoulli OPD substantially restores optimization signal to high-confidence tokens while remaining sensitive to teacher-student disagreement. We further analyze the resulting model behavior through confidence distributions, answer calibration, policy movement, and training dynamics, showing that Bernoulli OPD's gains cannot be explained simply by longer reasoning, larger optimization steps, or more uniform imitation of the teacher. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.02093 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.543Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).