Metraipolis › archive › dc_4116eb439e7f
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation — arXiv:2610.28194
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.28194
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-21T07:08:19.017Z
- SHA-256 of the archived text
85a575681a62c4e772fa2d18c3db6367243ad3fb44ba83efb7d29a59de9517a3- Stable URL of this page
https://metraipolis.com/documento/dc_4116eb439e7f
How to cite
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation — arXiv:2610.28194 — arXiv (en), https://arxiv.org/abs/2610.28194 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4116eb439e7f (Metraipolis archive dc_4116eb439e7f, sha256 85a575681a62c4e772fa2d18c3db6367243ad3fb44ba83efb7d29a59de9517a3)
BibTeX
@misc{dc4116eb439e7f,
title = {Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation — arXiv:2610.28194},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.28194},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4116eb439e7f; sha256 85a575681a62c4e772fa2d18c3db6367243ad3fb44ba83efb7d29a59de9517a3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.28194. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) supervises student-generated responses using sampled-token student-teacher log-ratios. A positive log-ratio penalizes a token that the student assigns higher probability to than the teacher, but its current value does not distinguish sustained high penalties from abrupt increases along a response. We examine these local changes by comparing consecutive blocks of tokens within the same response. Our diagnostic identifies penalty jumps, abrupt increases in average penalty that persist on held-out tokens. Across these jumps, penalties rise while teacher confidence decreases. We call this pattern penalty-confidence tension. Boundaries into high-penalty blocks do not exhibit comparable confidence changes, showing that local changes provide information beyond the current penalty value. Motivated by this observation, we propose Adaptive Pullback OPD (AP-OPD). At each detected jump, AP-OPD uses the preceding block as a local reference and uniformly scales all supervision coefficients in the following block so that its weighted average penalty matches the preceding block's original mean. This reference determines the scaling factor without an additional strength constant. AP-OPD improves performance on both mathematical reasoning and code generation. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.28194 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-21T07:05:43.630Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).