Metraipolis › archive › dc_654293ed0bd3

Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random — arXiv:2609.00576

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00576
License (as archived)
CC0-1.0
Archived
2026-10-17T06:31:58.093Z
SHA-256 of the archived text
d152260f86e413fcec5cfe54c18b87267ce594aee3c9e382733746ca70c19bd5
Stable URL of this page
https://metraipolis.com/documento/dc_654293ed0bd3

How to cite

Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random — arXiv:2609.00576 — arXiv (en), https://arxiv.org/abs/2609.00576 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_654293ed0bd3 (Metraipolis archive dc_654293ed0bd3, sha256 d152260f86e413fcec5cfe54c18b87267ce594aee3c9e382733746ca70c19bd5)
BibTeX
@misc{dc654293ed0bd3,
  title = {Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random — arXiv:2609.00576},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00576},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_654293ed0bd3; sha256 d152260f86e413fcec5cfe54c18b87267ce594aee3c9e382733746ca70c19bd5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00576. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Item-sensitivity, defined as whether a model's choice depends on the specific input rather than on its own output prior, is widely reported as evidence of task competence. We show this evidence is necessary but not sufficient using a forced-choice signalling task abstracted from the board game Deception: Murder in Hong Kong. In this environment, the reference points against which a coordinate should be judged (a fit-maximising strategy, a posterior-maximising strategy, and uniform random selection) are all computable in closed form. Across seven language models, two model families, a post-training ablation, and three independent scoring rules, every one of 21 model-by-rule cells is reliably item-sensitive. Yet 8 of those 21 cells are not statistically distinguishable from a chooser that ignores the item and selects at random, and 5 score worse than random at describing the target. Item-sensitivity and distance from random correlate at only r = 0.30. We call this consistency without alignment and argue it generalises to any evaluation that relies on item-sensitivity, permutation consistency, or self-consistency without an independent reference for the measured quantity. We further find that a literal-similarity baseline with no pragmatics outperforms most tested language models, that adding a pragmatic layer over two baseline similarity sources moves choosers toward random rather than toward the Bayesian reference, and that a standard labelled multiple-choice format carries no measurable content signal here. All results represent the model side of a pre-registered instrument; a matched human condition is designed and piloted but not yet collected.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.00576 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:15:12.619Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).