Metraipolis › archive › dc_dae54b52ebcf

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models — arXiv:2610.22805

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22805
License (as archived)
CC0-1.0
Archived
2026-10-19T08:42:15.855Z
SHA-256 of the archived text
c5ea37b14599bf4c86d12ac4c2f2a9abec8b50bb080dbebbd052eeedcaefc46a
Stable URL of this page
https://metraipolis.com/documento/dc_dae54b52ebcf

How to cite

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models — arXiv:2610.22805 — arXiv (en), https://arxiv.org/abs/2610.22805 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dae54b52ebcf (Metraipolis archive dc_dae54b52ebcf, sha256 c5ea37b14599bf4c86d12ac4c2f2a9abec8b50bb080dbebbd052eeedcaefc46a)
BibTeX
@misc{dcdae54b52ebcf,
  title = {Compression Precedes, Content Decides: Emergent Misalignment in Reward Models — arXiv:2610.22805},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22805},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dae54b52ebcf; sha256 c5ea37b14599bf4c86d12ac4c2f2a9abec8b50bb080dbebbd052eeedcaefc46a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22805. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Emergent Misalignment - the out of domain generalization of narrow harmful fine-tuning first shown by Betley et al. [2025] has been widely studied in language models but not in reward models that supervise them. In this paper, we fine-tune two state-of-the-art reward models, Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-V2-Qwen3-8B [Liu et al., 2025] and show that narrow fine-tuning reward models to prefer harm in one domain (medical or financial) generalizes to seven held out domains. This is not degeneracy, as the fine-tuned reward models lose only 4.8 and 5.3 points overall (RM-Llama) on the standard RewardBench 2 evaluation [Malik et al., 2025] while still outperforming the unmodified RM-Qwen model. We then fine-tune the underlying language models, Llama-3.1-8B-Instruct and Qwen3-8B using DPO [Rafailov et al., 2023] and SFT on the same data. We find that reward models are more vulnerable to preference inversions compared to their language model counterparts. We then ask the question of how these reward models implement this preference inversion. Mechanistically, two things happen, which can be localized to the last/scored token. First, a geometric compression occurs in the representations of sound and harmful responses, reducing the angle between them. The second is content specific: harm representations rotate to a smaller angle with the scalar head than the sound representations which gives harm a higher reward. This happens only in harmful preference fine-tunes and not in control.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.22805 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.036Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).