Metraipolis › archive › dc_bf61289b3c71

Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable — arXiv:2610.30349

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30349
License (as archived)
CC0-1.0
Archived
2026-10-25T08:33:51.073Z
SHA-256 of the archived text
90e704c5162c932e4f1c2c0d95b4e84cc7d23fc683eeaf1dec7826f5a4735d74
Stable URL of this page
https://metraipolis.com/documento/dc_bf61289b3c71

How to cite

Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable — arXiv:2610.30349 — arXiv (en), https://arxiv.org/abs/2610.30349 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf61289b3c71 (Metraipolis archive dc_bf61289b3c71, sha256 90e704c5162c932e4f1c2c0d95b4e84cc7d23fc683eeaf1dec7826f5a4735d74)
BibTeX
@misc{dcbf61289b3c71,
  title = {Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable — arXiv:2610.30349},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30349},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf61289b3c71; sha256 90e704c5162c932e4f1c2c0d95b4e84cc7d23fc683eeaf1dec7826f5a4735d74},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30349. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well. We train a reinforcement learning policy initialized from a state-of-the-art language model in a self-judged tournament with masked thinking: the generated arguments are judged pairwise on argument quality by the policy itself. As an empirical basis, we extract authentic dilemmas from classic novels, using a logic solver to verify that each constitutes a conflict. We show that our judge rates argument quality like human annotators, and the policy then adopts the language register that it rewards. Testing on the external benchmarks MoralChoice, ELEPHANT, and MORABLES, we find that the policy becomes more decisive and invariant to the order in which options are presented, but sycophancy and reframing remain a challenge to validity. Two ablation studies with synthetic dilemmas and unmasked thinking match the reordering gains, but show a weaker register shift and higher sycophancy. We conclude that consistency is not validity: consistency under reordering can be optimized without improving the moral judgment it is meant to signal.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.30349 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:13.141Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).