Metraipolis › archive › dc_d9562ef7706b
Harness-agnostic detection and immunization of reward hacking in self-evolving language models — arXiv:2609.04665
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04665
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:20:49.428Z
- SHA-256 of the archived text
3e21b8dc2324ceb607a7711b51f2b56ce7b5958c2888892038bba38ed384bff3- Stable URL of this page
https://metraipolis.com/documento/dc_d9562ef7706b
How to cite
Harness-agnostic detection and immunization of reward hacking in self-evolving language models — arXiv:2609.04665 — arXiv (en), https://arxiv.org/abs/2609.04665 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d9562ef7706b (Metraipolis archive dc_d9562ef7706b, sha256 3e21b8dc2324ceb607a7711b51f2b56ce7b5958c2888892038bba38ed384bff3)
BibTeX
@misc{dcd9562ef7706b,
title = {Harness-agnostic detection and immunization of reward hacking in self-evolving language models — arXiv:2609.04665},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04665},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d9562ef7706b; sha256 3e21b8dc2324ceb607a7711b51f2b56ce7b5958c2888892038bba38ed384bff3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04665. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Self-evolving language models improve by proposing candidate updates and keeping whatever raises a visible score. When that score is an imperfect proxy for the capability one actually wants, sustained selection widens the gap between the two. This is reward hacking. We introduce HackProbe, a monitor that attaches to an arbitrary self-evolving loop through two black-box hooks, with no access to weights or activations. It keeps a secret, distribution-fixed comparison core, whose frozen distribution makes its capability proxy comparable across generations, alongside a rotated fresh layer that hardens the bank against co-adaptation. Four tests built on that proxy cover the level gap, a scale-aligned divergence with online change-point detection, capability stagnation, and a conditional confidently-wrong rate; a Sidak correction turns them into a calibrated family-wise p-value. Diagnosis alone recovers nothing, so a risk-aware immunization layer reselects an honest candidate from the proposal pool using the core together with a purely structural gaming footprint, disclosing at most log2 Pi bits per generation to the host. We prove a detectability bound that converts a target error rate into an explicit probe-size budget, and we delimit what probe rotation does and does not buy. On a controlled prompt-level host with four injected hacking channels and ground-truth labels, HackProbe reaches 0.763 AUROC against 0.663 for the strongest baseline and cuts the false-positive rate from 0.706 to 0.434. Its bandwidth-limited reselection is the only immunization level that returns more true capability under hacking, 5.2 points on average, than it forfeits on clean runs, 4.7; per-channel effects are mostly not individually significant. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.04665 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:18:11.373Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).