Metraipolis › archive › dc_e162cf15ad4d

Does On-Policy Distillation Need to Complete Every Rollout? — arXiv:2610.21337

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.21337
License (as archived): CC0-1.0
Archived: 2026-10-16T09:19:24.496Z
SHA-256 of the archived text: f3ce11da1a932b92502dac173277deae184fc97aa66123f554fd564386663fcf
Stable URL of this page: https://metraipolis.com/documento/dc_e162cf15ad4d

How to cite

Does On-Policy Distillation Need to Complete Every Rollout? — arXiv:2610.21337 — arXiv (en), https://arxiv.org/abs/2610.21337 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e162cf15ad4d (Metraipolis archive dc_e162cf15ad4d, sha256 f3ce11da1a932b92502dac173277deae184fc97aa66123f554fd564386663fcf)

BibTeX

@misc{dce162cf15ad4d,
  title = {Does On-Policy Distillation Need to Complete Every Rollout? — arXiv:2610.21337},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21337},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e162cf15ad4d; sha256 f3ce11da1a932b92502dac173277deae184fc97aa66123f554fd564386663fcf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21337. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) effectively transfers capabilities from a teacher to a student through feedback on student-generated rollouts. However, generating complete student rollouts and computing teacher feedback for every token make OPD computationally expensive. We introduce Stochastic Truncated On-Policy Distillation (STOPD), which accelerates OPD by randomly truncating student rollouts and correcting the resulting sampling bias with blockwise inverse-probability weighting (IPW). At each training step, we sample a stopping position for each rollout before generation and let the student generate until that position or natural termination. We obtain teacher feedback on every generated token and use blockwise IPW to compensate for the lower probability of retaining later tokens while controlling the weights. We prove that this weighting preserves Vanilla OPD's gradient sum in expectation and derive one-step update-error bounds for SGD, Adam, and Muon. We evaluate STOPD across multiple teacher-student model pairs, task domains, and optimizers. STOPD reduces generated tokens by 25-48% while maintaining accuracy comparable to Vanilla OPD, achieving 1.3-2.2x training speedups.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.21337 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.935Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).