Metraipolis › archive › dc_87ae71cc3610
O$^2$PD: Overlap-Gated On-Policy Distillation on Reliable Prefix — arXiv:2611.07731
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.07731
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:42:37.251Z
- SHA-256 of the archived text
5f3c0e38319a191fba55eef60df451f5baff0bc526ae3d7582c773cb4a165310- Stable URL of this page
https://metraipolis.com/documento/dc_87ae71cc3610
How to cite
O$^2$PD: Overlap-Gated On-Policy Distillation on Reliable Prefix — arXiv:2611.07731 — arXiv (en), https://arxiv.org/abs/2611.07731 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_87ae71cc3610 (Metraipolis archive dc_87ae71cc3610, sha256 5f3c0e38319a191fba55eef60df451f5baff0bc526ae3d7582c773cb4a165310)
BibTeX
@misc{dc87ae71cc3610,
title = {O$^2$PD: Overlap-Gated On-Policy Distillation on Reliable Prefix — arXiv:2611.07731},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.07731},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_87ae71cc3610; sha256 5f3c0e38319a191fba55eef60df451f5baff0bc526ae3d7582c773cb4a165310},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07731. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) supplies dense token-level supervision by aligning a student to the teacher's next-token distribution on its own generated rollouts. Existing works mostly implement top-$k$ OPD, applying reverse Kullback-Leibler ($\RKL$) divergence between the teacher's and the student's renormalized distributions over teacher top-$k$ support. However, the teacher scores student-generated rollouts that are inherently off-policy for itself, leaving the reliability of its next-token guidance unclear. Restricting the divergence to the teacher top-$k$ support also discards the remaining mass outside that support. These gaps leave two questions implicit: 1) which tokens to align, and 2) how to align those tokens. To answer these questions, we propose Overlap-gated On-Policy Distillation (O$^{2}$PD). As for 1), we quantitatively analyze the reliability of teacher supervision and find that alignment on the student's trajectory prefix is more reliable than alignment on the full context. Therefore, our O$^{2}$PD is applied only to the 20\% prefix tokens of each student rollout. As for 2), we theoretically characterize two problems of top-$k$ OPD: the \emph{Teacher-support Uncovering} and the \emph{Dominant-Mode Mismatch}. Based on these analyses, we propose two support strategies: the \emph{Reset Support} strategy attaches a reset atom to every token, pooling mass outside the teacher top-$k$, and the \emph{Support Convergence} strategy converges the support to the already aligned intersection. Our O$^{2}$PD employs the designed overlap-gated $\alpha$-divergence to interpolate between mode-covering and mode-seeking for per-token alignment, where the overlap is used as a gate to switch the support and adaptively modulate $\alpha$.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.07731 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:42.687Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).