Metraipolis › archive › dc_efb87f7a9d06
RM-OPD: On-Policy Distillation for Generative Reward Models — arXiv:2610.20174
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20174
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T14:19:43.785Z
- SHA-256 of the archived text
8edcd0add1ec487a04569d793744e99dc7ec756f861809cb5638408f40be9617- Stable URL of this page
https://metraipolis.com/documento/dc_efb87f7a9d06
How to cite
RM-OPD: On-Policy Distillation for Generative Reward Models — arXiv:2610.20174 — arXiv (en), https://arxiv.org/abs/2610.20174 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_efb87f7a9d06 (Metraipolis archive dc_efb87f7a9d06, sha256 8edcd0add1ec487a04569d793744e99dc7ec756f861809cb5638408f40be9617)
BibTeX
@misc{dcefb87f7a9d06,
title = {RM-OPD: On-Policy Distillation for Generative Reward Models — arXiv:2610.20174},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20174},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_efb87f7a9d06; sha256 8edcd0add1ec487a04569d793744e99dc7ec756f861809cb5638408f40be9617},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20174. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.20174 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:53:35.708Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).