Metraipolis › archive › dc_92ecef374ce8

How to post-train on a surrogate: Envelope sampling mitigates reward hacking — arXiv:2610.11281

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11281
License (as archived)
CC0-1.0
Archived
2026-10-09T05:57:28.310Z
SHA-256 of the archived text
9f26ccfb6c126a5d9ee1167f711e064179623cc6d0f941c99a2849c9412ca6c7
Stable URL of this page
https://metraipolis.com/documento/dc_92ecef374ce8

How to cite

How to post-train on a surrogate: Envelope sampling mitigates reward hacking — arXiv:2610.11281 — arXiv (en), https://arxiv.org/abs/2610.11281 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_92ecef374ce8 (Metraipolis archive dc_92ecef374ce8, sha256 9f26ccfb6c126a5d9ee1167f711e064179623cc6d0f941c99a2849c9412ca6c7)
BibTeX
@misc{dc92ecef374ce8,
  title = {How to post-train on a surrogate: Envelope sampling mitigates reward hacking — arXiv:2610.11281},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11281},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_92ecef374ce8; sha256 9f26ccfb6c126a5d9ee1167f711e064179623cc6d0f941c99a2849c9412ca6c7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11281. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable side effects. In this work, we study a setting in which a small number $n$ of model outputs are annotated with ground-truth labels (e.g., from expert review) and used to recalibrate the LLM judge before optimizing against it. Prior approaches to judge recalibration are costly or heuristic, and it is known that on-policy sampling fails when the surrogate is miscalibrated on a rare set of outputs. In this work, we propose envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an $L^2$ ball around the judge. We give practical algorithms to sample from the envelope by rejection or by fine-tuning against a modified reward, and experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking where recalibrating on base-model samples does not.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Methodology (stat.ME).
Pagina del lavoro: https://arxiv.org/abs/2610.11281 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.601Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).