Metraipolis › archive › dc_f587f22fa67a

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment — arXiv:2609.37914

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37914
License (as archived)
CC0-1.0
Archived
2026-10-07T16:27:54.248Z
SHA-256 of the archived text
b3dcccbb77f46c27a41fe9799c712e09adde10012282e6ce9a6122e5df62d27d
Stable URL of this page
https://metraipolis.com/documento/dc_f587f22fa67a

How to cite

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment — arXiv:2609.37914 — arXiv (en), https://arxiv.org/abs/2609.37914 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f587f22fa67a (Metraipolis archive dc_f587f22fa67a, sha256 b3dcccbb77f46c27a41fe9799c712e09adde10012282e6ce9a6122e5df62d27d)
BibTeX
@misc{dcf587f22fa67a,
  title = {The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment — arXiv:2609.37914},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37914},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f587f22fa67a; sha256 b3dcccbb77f46c27a41fe9799c712e09adde10012282e6ce9a6122e5df62d27d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37914. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as \emph{emergent misalignment} (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.37914 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:54.048Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).