Metraipolis › archive › dc_ef694dfdc873

Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning — arXiv:2611.01183

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01183
License (as archived): CC0-1.0
Archived: 2026-11-03T11:05:44.178Z
SHA-256 of the archived text: c26396128f54c17a8553536d6ddb8a35b73d039ecda12f4328b21c6abdae4b1b
Stable URL of this page: https://metraipolis.com/documento/dc_ef694dfdc873

How to cite

Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning — arXiv:2611.01183 — arXiv (en), https://arxiv.org/abs/2611.01183 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef694dfdc873 (Metraipolis archive dc_ef694dfdc873, sha256 c26396128f54c17a8553536d6ddb8a35b73d039ecda12f4328b21c6abdae4b1b)

BibTeX

@misc{dcef694dfdc873,
  title = {Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning — arXiv:2611.01183},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01183},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef694dfdc873; sha256 c26396128f54c17a8553536d6ddb8a35b73d039ecda12f4328b21c6abdae4b1b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01183. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Bounded intrinsic rewards are often assumed to be safe from exploitation: if a signal can yield at most a fixed amount, gaming it should not pay. We test this for a homeostatic satiation reward, in which needs build up and relieving them is rewarding, and give the agent a cheap action that resets its needs. In MiniGrid DoorKey-8x8, PPO agents with this action solved 2 of 80 seeds, against 71 of 80 on the same seeds without it; in KeyCorridorS3R1, 0 of 20 against 20 of 20. Their policies stayed near uniform. The reward does not ask for this: a telescoping argument caps any policy's satiation return at the needs' starting drive, and a post-hoc computation shows that, at every growth rate we trained, a solving trajectory beats that cap on all sampled layouts. In DoorKey with advantage normalization on, a satisfaction signal the agent could fill itself was worse than none (11 of 20 solved with task reward alone, 1 of 20 with the tamperable signal), and a per-press cost of 0.1 restored the task in every seed. Farming of positive reward cycles appeared in DoorKey but not persistently in KeyCorridorS3R1. All predictions were pre-registered, and hash-locked from round 3 on; 9 of our 23 hypotheses were not supported, and we report each.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01183 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:45.994Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).