Metraipolis › archive › dc_f47a8ab778aa
On-policy Distillation with Verifiable Reward — arXiv:2608.24696
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.24696
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:36:21.419Z
- SHA-256 of the archived text
7e51e2d6c06e434be0b1e522f150c28d9ef5bddb0a7d9ebbf288efb408791bfc- Stable URL of this page
https://metraipolis.com/documento/dc_f47a8ab778aa
How to cite
On-policy Distillation with Verifiable Reward — arXiv:2608.24696 — arXiv (en), https://arxiv.org/abs/2608.24696 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f47a8ab778aa (Metraipolis archive dc_f47a8ab778aa, sha256 7e51e2d6c06e434be0b1e522f150c28d9ef5bddb0a7d9ebbf288efb408791bfc)
BibTeX
@misc{dcf47a8ab778aa,
title = {On-policy Distillation with Verifiable Reward — arXiv:2608.24696},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.24696},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f47a8ab778aa; sha256 7e51e2d6c06e434be0b1e522f150c28d9ef5bddb0a7d9ebbf288efb408791bfc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24696. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.24696 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:28:52.269Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).