Metraipolis › archive › dc_4e597795bb60

Measuring the Assistant's Harmlessness Preferences on the User Turn — arXiv:2609.23935

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.23935
License (as archived): CC0-1.0
Archived: 2026-10-09T06:49:58.507Z
SHA-256 of the archived text: 6a6dbfc1a3a9f2a5c86b483bf5423c2ff60668de18a752a19cc118f4036cc57d
Stable URL of this page: https://metraipolis.com/documento/dc_4e597795bb60

How to cite

Measuring the Assistant's Harmlessness Preferences on the User Turn — arXiv:2609.23935 — arXiv (en), https://arxiv.org/abs/2609.23935 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e597795bb60 (Metraipolis archive dc_4e597795bb60, sha256 6a6dbfc1a3a9f2a5c86b483bf5423c2ff60668de18a752a19cc118f4036cc57d)

BibTeX

@misc{dc4e597795bb60,
  title = {Measuring the Assistant's Harmlessness Preferences on the User Turn — arXiv:2609.23935},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23935},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e597795bb60; sha256 6a6dbfc1a3a9f2a5c86b483bf5423c2ff60668de18a752a19cc118f4036cc57d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23935. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training turns a general next-token predictor into a chat model with a persistent assistant persona. If that persona is a character the model plays only on its own turns, its preferences should govern what the assistant says, not what the model predicts other speakers will say. We test this boundary and find that it does not hold: a safety-relevant preference of the assistant---for harmless over harmful tasks---shapes the model's predictions even on the user's turn, where the assistant is not the one speaking. We find that this preference is small or near-zero in pretrained base models, that it emerges through post-training, replicated across open-weight model families, grows with scale, and can be moved by narrow finetuning that never touches user turns. We claim that this is evidence that post-training does not merely install a shallow assistant persona, but instead generalises beyond just the local assistant turn, into the model's representation of the user.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.23935 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:45.016Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).