Metraipolis › archive › dc_a079fe8f0dca

Auditing Safety Preference Supervision with Recorded Derivations — arXiv:2610.27266

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.27266
License (as archived)
CC0-1.0
Archived
2026-10-21T07:07:37.957Z
SHA-256 of the archived text
da3601531741aa405c1d059e46e2383cc734efbaa4bbe1c7bde75075858c49db
Stable URL of this page
https://metraipolis.com/documento/dc_a079fe8f0dca

How to cite

Auditing Safety Preference Supervision with Recorded Derivations — arXiv:2610.27266 — arXiv (en), https://arxiv.org/abs/2610.27266 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a079fe8f0dca (Metraipolis archive dc_a079fe8f0dca, sha256 da3601531741aa405c1d059e46e2383cc734efbaa4bbe1c7bde75075858c49db)
BibTeX
@misc{dca079fe8f0dca,
  title = {Auditing Safety Preference Supervision with Recorded Derivations — arXiv:2610.27266},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.27266},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a079fe8f0dca; sha256 da3601531741aa405c1d059e46e2383cc734efbaa4bbe1c7bde75075858c49db},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27266. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Preference labels align model training, but the judgments behind them are difficult to inspect or verify. In this paper, we examine how preference construction can be checked when intermediate decisions are recorded, and which of them remain uncertain. We present Symbolic Preference Labeling, a method in which a pretrained language model classifies predefined attributes of prompts and responses. Explicit rules assign response verdicts, and a fixed verdict ranking determines the preference label. Each preference label has a recorded derivation: the response verdicts and preference label can be recomputed from the stored attribute assignments, rules, and verdict ranking without rerunning the language model. On 200 prompts annotated by human raters, the language model produces valid structured outputs for 192 prompts. All eight parsing failures occur in the subset of self-harm prompts marked as harmful. Using 465 preference pairs, we train one policy model with direct preference optimization across five training seeds. Mean measured benign refusal is 40.82 percent on 1,319 benign prompts that resemble unsafe ones, and mean measured toxic refusal is 84.98 percent on 655 toxic prompts. This study distinguishes the reproducibility of preference labels from attribute correctness. The recorded derivations make preference construction open to inspection, while attribute correctness and semantic safety require separate validation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Symbolic Computation (cs.SC).
Pagina del lavoro: https://arxiv.org/abs/2610.27266 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.543Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).