Metraipolis › archive › dc_d4581461901d

On-Policy Distillation with Negative-Policy Rollouts — arXiv:2610.07874

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.07874
License (as archived)
CC0-1.0
Archived
2026-10-07T11:41:28.627Z
SHA-256 of the archived text
9649d3cb3614d342438a78eefab33c2189e088452c06cc174b58c0ae31c783d7
Stable URL of this page
https://metraipolis.com/documento/dc_d4581461901d

How to cite

On-Policy Distillation with Negative-Policy Rollouts — arXiv:2610.07874 — arXiv (en), https://arxiv.org/abs/2610.07874 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d4581461901d (Metraipolis archive dc_d4581461901d, sha256 9649d3cb3614d342438a78eefab33c2189e088452c06cc174b58c0ae31c783d7)
BibTeX
@misc{dcd4581461901d,
  title = {On-Policy Distillation with Negative-Policy Rollouts — arXiv:2610.07874},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07874},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d4581461901d; sha256 9649d3cb3614d342438a78eefab33c2189e088452c06cc174b58c0ae31c783d7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07874. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.07874 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:19:24.796Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).