Metraipolis › archive › dc_dfc8e1295785

Scaling Laws for Generative Reward Models — arXiv:2610.30765

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30765
License (as archived)
CC0-1.0
Archived
2026-10-23T15:13:02.408Z
SHA-256 of the archived text
21d51e7a71a51b1bb4362e5b260595419914763e6e97a4be1d73200e952ab3b2
Stable URL of this page
https://metraipolis.com/documento/dc_dfc8e1295785

How to cite

Scaling Laws for Generative Reward Models — arXiv:2610.30765 — arXiv (en), https://arxiv.org/abs/2610.30765 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dfc8e1295785 (Metraipolis archive dc_dfc8e1295785, sha256 21d51e7a71a51b1bb4362e5b260595419914763e6e97a4be1d73200e952ab3b2)
BibTeX
@misc{dcdfc8e1295785,
  title = {Scaling Laws for Generative Reward Models — arXiv:2610.30765},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30765},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dfc8e1295785; sha256 21d51e7a71a51b1bb4362e5b260595419914763e6e97a4be1d73200e952ab3b2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30765. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.30765 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:08.397Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).