Metraipolis › archive › dc_56b59a696a4d

Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking — arXiv:2609.14998

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.14998
License (as archived)
CC0-1.0
Archived
2026-10-15T22:47:44.300Z
SHA-256 of the archived text
76c2535ba1accc16467683c79abfd5f8629d0033cd597f11559532d2ec2f6b27
Stable URL of this page
https://metraipolis.com/documento/dc_56b59a696a4d

How to cite

Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking — arXiv:2609.14998 — arXiv (en), https://arxiv.org/abs/2609.14998 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_56b59a696a4d (Metraipolis archive dc_56b59a696a4d, sha256 76c2535ba1accc16467683c79abfd5f8629d0033cd597f11559532d2ec2f6b27)
BibTeX
@misc{dc56b59a696a4d,
  title = {Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking — arXiv:2609.14998},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.14998},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_56b59a696a4d; sha256 76c2535ba1accc16467683c79abfd5f8629d0033cd597f11559532d2ec2f6b27},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14998. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.14998 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.246Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).