Metraipolis › archive › dc_64038d137197

Outcome-Guided On-Policy Self-Distillation — arXiv:2610.05070

Source
arXiv
Original
https://arxiv.org/abs/2610.05070
License (as archived)
CC0-1.0
Archived
2026-10-06T05:30:53.438Z
SHA-256 of the archived text
0c3668985e99f291201cdf6518f216b3a19f79649883839c83e2e9a82d3fab68
Stable URL of this page
https://metraipolis.com/documento/dc_64038d137197

How to cite

Outcome-Guided On-Policy Self-Distillation — arXiv:2610.05070 — arXiv, https://arxiv.org/abs/2610.05070 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_64038d137197 (Metraipolis archive dc_64038d137197, sha256 0c3668985e99f291201cdf6518f216b3a19f79649883839c83e2e9a82d3fab68)
BibTeX
@misc{dc64038d137197,
  title = {Outcome-Guided On-Policy Self-Distillation — arXiv:2610.05070},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05070},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_64038d137197; sha256 0c3668985e99f291201cdf6518f216b3a19f79649883839c83e2e9a82d3fab68},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05070. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy self-distillation (OPSD) provides denser token-level supervision and better computational efficiency than Reinforcement Learning with Verifiable Rewards (RLVR). However, this denser supervision may introduce substantial noise and training instability. Existing improvements often rely on high-variance per-token statistics and introduce extra hyperparameters and trade-offs. Based on the advantage formulation in RLVR, we analyze the OPSD objective from the same perspective, incorporating outcome correctness signals. We find that vanilla OPSD imposes insufficient penalties and excessive rewards on incorrect trajectories because it applies a fixed divergence objective regardless of outcome correctness. Furthermore, the reliability of teacher supervision is associated with both trajectory outcome and the cumulative average teacher entropy along the rollout. Based on these observations, we propose Outcome-Guided On-Policy Self-Distillation (OG-OPSD), which dynamically adapts both the divergence objective and distillation position according to binary outcome rewards and the cumulative average teacher entropy. Extensive experiments show that OG-OPSD consistently improves the performance of vanilla OPSD and multiple strong baselines in mathematical reasoning, multimodal reasoning, and out-of-distribution tasks across Qwen3 models at 1.7B, 4B, and 8B scales, as well as Qwen3-VL-2B.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.05070 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.761Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).