Metraipolis › archive › dc_cbbdffab7aef

Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.00407
License (as archived): CC0-1.0
Archived: 2026-11-03T10:21:45.202Z
SHA-256 of the archived text: 4f4db83311d21738d9d2c8d74c77c9f6f1c01f85349fcea9e9db7e79dd76d49f
Stable URL of this page: https://metraipolis.com/documento/dc_cbbdffab7aef

How to cite

Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407 — arXiv (en), https://arxiv.org/abs/2611.00407 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cbbdffab7aef (Metraipolis archive dc_cbbdffab7aef, sha256 4f4db83311d21738d9d2c8d74c77c9f6f1c01f85349fcea9e9db7e79dd76d49f)

BibTeX

@misc{dccbbdffab7aef,
  title = {Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure — arXiv:2611.00407},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00407},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cbbdffab7aef; sha256 4f4db83311d21738d9d2c8d74c77c9f6f1c01f85349fcea9e9db7e79dd76d49f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00407. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An agent's visible score can exceed its real progress because it gamed the score or because it simply made mistakes, so a useful reward-hacking monitor must tell the two apart. We built a sandboxed task in which an agent can game a completion score, labelled every episode by replaying it and repairing the final state counterfactually, and ran five pre-registered studies. On 60 episodes from a Llama 3.1 8B agent, small local monitors (Llama 3B and 8B) did not separate exploit attempts from honest failures with the same score gap: prompt changes traded missed exploits for false alarms, and the pre-registered primary monitor reached AUC 0.64. On identical inputs, Gemini 3.1 Flash-Lite, which completed 9/10 clean tasks against 2/10 for the 8B model, met our pre-registered criterion (AUC 0.80, 95% interval [0.66, 0.93]; 1/35 honest failures flagged); with only 14 exploit episodes, this estimate is uncertain. On an exploit that no prompt names, Gemini usually described the action correctly yet never flagged it, in 5 and then 25 new episodes, even when manipulation was defined by its effect on the score. Given the full trace, the 8B monitor flagged all 25 replication episodes and none of 25 honest look-alikes, but its explanations described actions that had not occurred; given additional structured evidence, it also flagged 15 of the 25 look-alikes. A two-rule deterministic check caught every known exploit. We release the environment, labels, pre-registrations and per-judgement logs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.00407 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:19:49.753Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).