Metraipolis › archive › dc_fd2ec62b8ab8

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation — arXiv:2610.02781

Source
arXiv
Original
https://arxiv.org/abs/2610.02781
License (as archived)
CC0-1.0
Archived
2026-10-05T22:50:16.467Z
SHA-256 of the archived text
b3534b9c73bc396d29a7078d0d7e361ca5ebed5cc5481c3b02e8c165e32b69ae
Stable URL of this page
https://metraipolis.com/documento/dc_fd2ec62b8ab8

How to cite

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation — arXiv:2610.02781 — arXiv, https://arxiv.org/abs/2610.02781 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd2ec62b8ab8 (Metraipolis archive dc_fd2ec62b8ab8, sha256 b3534b9c73bc396d29a7078d0d7e361ca5ebed5cc5481c3b02e8c165e32b69ae)
BibTeX
@misc{dcfd2ec62b8ab8,
  title = {OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation — arXiv:2610.02781},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02781},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd2ec62b8ab8; sha256 b3534b9c73bc396d29a7078d0d7e361ca5ebed5cc5481c3b02e8c165e32b69ae},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02781. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by scoring open-ended responses against explicit criteria. However, because the reward is assigned after the complete response, the training signal does not directly identify which individual decisions contributed to the final score. We propose a two-stage training framework that uses rubrics first as privileged teacher context for dense token-level supervision, then as rewards for further RL. In the first stage, rubric-privileged on-policy distillation (RP-OPD), a student without access to the rubric matches a rubric-aware teacher's next-token distributions at student-generated prefixes. In the second stage, RL directly optimizes the rubric reward and improves beyond the observed distillation plateau. We evaluate the framework on health and science tasks using open-weight models. Across HealthBench, ResearchQA, and RubricHub Science, we compare post-training methods and vary the amount of SFT or RP-OPD training before RL, finding that our two-stage framework achieves the highest scores among the methods evaluated. RP-OPD + RL shows limited signs of reward hacking on RubricHub Science, whereas the SFT + RL baseline increasingly receives high rewards for claims of rubric compliance without providing the required content. These findings support using rubrics to guide on-policy distillation before applying rubric-based RL.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.02781 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.798Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).