Metraipolis › archive › dc_5cc2186e6285

Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00407
License (as archived)
CC0-1.0
Archived
2026-11-03T11:00:42.591Z
SHA-256 of the archived text
71d5f6bafbfa19fa0ee507bd4fd7ab869593a44aa85236f8d308891f358677a4
Stable URL of this page
https://metraipolis.com/documento/dc_5cc2186e6285

How to cite

Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407 — arXiv (en), https://arxiv.org/abs/2611.00407 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5cc2186e6285 (Metraipolis archive dc_5cc2186e6285, sha256 71d5f6bafbfa19fa0ee507bd4fd7ab869593a44aa85236f8d308891f358677a4)
BibTeX
@misc{dc5cc2186e6285,
  title = {Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00407},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5cc2186e6285; sha256 71d5f6bafbfa19fa0ee507bd4fd7ab869593a44aa85236f8d308891f358677a4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00407. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An agent's visible score can exceed its real progress because it gamed the score or because it simply made mistakes, so a useful reward-hacking monitor must tell the two apart. We built a sandboxed task in which an agent can game a completion score, labelled every episode by replaying it and repairing the final state counterfactually, and ran five pre-registered studies. On 60 episodes from a Llama 3.1 8B agent, small local monitors (Llama 3B and 8B) did not separate exploit attempts from honest failures with the same score gap: prompt changes traded missed exploits for false alarms, and the pre-registered primary monitor reached AUC 0.64. On identical inputs, Gemini 3.1 Flash-Lite, which completed 9/10 clean tasks against 2/10 for the 8B model, met our pre-registered criterion (AUC 0.80, 95% interval [0.66, 0.93]; 1/35 honest failures flagged); with only 14 exploit episodes, this estimate is uncertain. On an exploit that no prompt names, Gemini usually described the action correctly yet never flagged it, in 5 and then 25 new episodes, even when manipulation was defined by its effect on the score. Given the full trace, the 8B monitor flagged all 25 replication episodes and none of 25 honest look-alikes, but its explanations described actions that had not occurred; given additional structured evidence, it also flagged 15 of the 25 look-alikes. A two-rule deterministic check caught every known exploit. We release the environment, labels, pre-registrations and per-judgement logs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.00407 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:45.706Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).