Metraipolis › archive › dc_2c354cb76bea
Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL — arXiv:2611.09828
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.09828
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-09T22:41:44.039Z
- SHA-256 of the archived text
19a271b6a9c9d46e715ebd7076f55545cc9c052c02dc4242d5e46961d8bf6971- Stable URL of this page
https://metraipolis.com/documento/dc_2c354cb76bea
How to cite
Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL — arXiv:2611.09828 — arXiv (en), https://arxiv.org/abs/2611.09828 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c354cb76bea (Metraipolis archive dc_2c354cb76bea, sha256 19a271b6a9c9d46e715ebd7076f55545cc9c052c02dc4242d5e46961d8bf6971)
BibTeX
@misc{dc2c354cb76bea,
title = {Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL — arXiv:2611.09828},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.09828},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c354cb76bea; sha256 19a271b6a9c9d46e715ebd7076f55545cc9c052c02dc4242d5e46961d8bf6971},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09828. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited. To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination. We first focus on the coding environment, where we find that: (1) models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges; (2) monitors that leak more learning signal during RL suppress reward hacking but are more often evaded; and (3) including easier problems in training can decrease reward hacking. We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.09828 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-09T22:33:47.165Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).