Metraipolis › archive › dc_790ea219e862

On the Off-Policy Teacher in On-Policy Distillation — arXiv:2609.38360

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38360
License (as archived)
CC0-1.0
Archived
2026-10-07T12:43:35.369Z
SHA-256 of the archived text
9f7c534a4eda8f8066902d0bcffd41f3058499098fed9f2fdf79f3b5c58027ce
Stable URL of this page
https://metraipolis.com/documento/dc_790ea219e862

How to cite

On the Off-Policy Teacher in On-Policy Distillation — arXiv:2609.38360 — arXiv (en), https://arxiv.org/abs/2609.38360 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_790ea219e862 (Metraipolis archive dc_790ea219e862, sha256 9f7c534a4eda8f8066902d0bcffd41f3058499098fed9f2fdf79f3b5c58027ce)
BibTeX
@misc{dc790ea219e862,
  title = {On the Off-Policy Teacher in On-Policy Distillation — arXiv:2609.38360},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38360},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_790ea219e862; sha256 9f7c534a4eda8f8066902d0bcffd41f3058499098fed9f2fdf79f3b5c58027ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38360. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.38360 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.863Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).