Metraipolis › archive › dc_a5555bc00a77

Forecasting How RLHF Responds to Reward-Model Edits Before Training — arXiv:2611.03244

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.03244
License (as archived): CC0-1.0
Archived: 2026-11-03T11:23:09.314Z
SHA-256 of the archived text: 84ff17b514fd47fc767d445fba3ca76c0bf408fcd2cd93c0c5c86fcfccdeb862
Stable URL of this page: https://metraipolis.com/documento/dc_a5555bc00a77

How to cite

Forecasting How RLHF Responds to Reward-Model Edits Before Training — arXiv:2611.03244 — arXiv (en), https://arxiv.org/abs/2611.03244 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a5555bc00a77 (Metraipolis archive dc_a5555bc00a77, sha256 84ff17b514fd47fc767d445fba3ca76c0bf408fcd2cd93c0c5c86fcfccdeb862)

BibTeX

@misc{dca5555bc00a77,
  title = {Forecasting How RLHF Responds to Reward-Model Edits Before Training — arXiv:2611.03244},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03244},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a5555bc00a77; sha256 84ff17b514fd47fc767d445fba3ca76c0bf408fcd2cd93c0c5c86fcfccdeb862},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03244. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve. Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. With counterfactual prices, the classical alternative, we explain 50%. The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.03244 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.685Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).