Metraipolis › archive › dc_982cf4729a75

Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision — arXiv:2609.37624

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37624
License (as archived)
CC0-1.0
Archived
2026-10-07T16:27:29.538Z
SHA-256 of the archived text
04c4c2e2a0239afe435c1188ed15ea1b41f0e0d70ee911f6fbb23aaa7ab10e20
Stable URL of this page
https://metraipolis.com/documento/dc_982cf4729a75

How to cite

Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision — arXiv:2609.37624 — arXiv (en), https://arxiv.org/abs/2609.37624 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_982cf4729a75 (Metraipolis archive dc_982cf4729a75, sha256 04c4c2e2a0239afe435c1188ed15ea1b41f0e0d70ee911f6fbb23aaa7ab10e20)
BibTeX
@misc{dc982cf4729a75,
  title = {Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision — arXiv:2609.37624},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37624},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_982cf4729a75; sha256 04c4c2e2a0239afe435c1188ed15ea1b41f0e0d70ee911f6fbb23aaa7ab10e20},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37624. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Fine-tuning a language model on a narrow set of harmful demonstrations, such as bad medical advice, can make it broadly misaligned on unrelated questions, a phenomenon known as emergent misalignment (EM). The usual defense is to find the offending rows and delete them, but a row locator failed our held-out test and deleting rows helps less than expected. We ask a different question: given a fixed set of poisoned rows, is it better to correct them than to remove them? We fine-tune Qwen2.5-14B-Instruct on a mixture of bad medical advice and benign chat data, select a quarter of the poison rows in advance, and either delete them or replace each with a corrected answer to the same prompt, keeping everything else the same. Replacing the rows cuts the EM rate by about a third and improves answers on held-out medical questions, while deleting the same rows has little measurable effect. The advantage is larger when half the poison rows are corrected, and it holds on a second base model and a second misaligned model organism. The content of the replacement appears to matter: paraphrasing the rows while keeping their bad advice shows no clear benefit, and the correct answers distributed with the dataset appear to do about as well as our rewriter's. Realigning an already-poisoned model with further fine-tuning is known to work, but which data does the work has not been compared directly. We find that a short round of training on corrections beats the same amount of training on generic chat data, that corrections on other medical prompts do roughly as well as corrections of the poisoned prompts themselves, and that instructing the correction writer to model a careful, harm-avoiding assistant adds no measurable benefit over plain corrections. In the settings we tested, correcting harmful training data reduces EM more than deleting it.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.37624 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:54.004Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).