Metraipolis › archive › dc_863068f796bc
Revisiting scaling laws for reward optimization — arXiv:2609.38526
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38526
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:43:53.112Z
- SHA-256 of the archived text
89b57af88d82c2c712a9fe107c7b7676ee6607d9290c3c3c8eda7563a72bd495- Stable URL of this page
https://metraipolis.com/documento/dc_863068f796bc
How to cite
Revisiting scaling laws for reward optimization — arXiv:2609.38526 — arXiv (en), https://arxiv.org/abs/2609.38526 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_863068f796bc (Metraipolis archive dc_863068f796bc, sha256 89b57af88d82c2c712a9fe107c7b7676ee6607d9290c3c3c8eda7563a72bd495)
BibTeX
@misc{dc863068f796bc,
title = {Revisiting scaling laws for reward optimization — arXiv:2609.38526},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38526},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_863068f796bc; sha256 89b57af88d82c2c712a9fe107c7b7676ee6607d9290c3c3c8eda7563a72bd495},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38526. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort---measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget, over-optimization (or reward hacking) can arise: because we optimize against a proxy reward model (distinct from true rewards), performance can plateau or degrade. Naturally, the proxy reward's accuracy depends on how much preference data (often in the form of pairwise comparisons) was used to train it. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context. Performance roughly scales as $\Theta(\sqrt{\min\{\log(M),K\}})$, where $M$ is the number of comparisons in training data and $K$ is the policy's divergence budget. We develop an information-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure. Informed by this, we conduct extensive empirical evaluations using a real-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models (0.6B to 4B). Our scaling law provides an excellent fit (R2 from 97\% to 99\%), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures (best-of-$N$ or policy tilting). Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38526 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.896Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).