Metraipolis › archive › dc_cc01d2b952b5

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases — arXiv:2608.10503

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.10503
License (as archived)
CC0-1.0
Archived
2026-10-19T06:14:12.091Z
SHA-256 of the archived text
91cf4e435be2bd27f50e70de5ac23637d5e15c597351bcb98701eb226741dacb
Stable URL of this page
https://metraipolis.com/documento/dc_cc01d2b952b5

How to cite

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases — arXiv:2608.10503 — arXiv (en), https://arxiv.org/abs/2608.10503 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cc01d2b952b5 (Metraipolis archive dc_cc01d2b952b5, sha256 91cf4e435be2bd27f50e70de5ac23637d5e15c597351bcb98701eb226741dacb)
BibTeX
@misc{dccc01d2b952b5,
  title = {Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases — arXiv:2608.10503},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10503},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cc01d2b952b5; sha256 91cf4e435be2bd27f50e70de5ac23637d5e15c597351bcb98701eb226741dacb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10503. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. The NLP community typically evaluates models using large, unstructured benchmarks. While effective for general capabilities, these datasets fundamentally conflate causal mechanisms: even when an aggregate bias is detected, unstructured evaluations cannot disentangle whether it stems from baseline traits, contextual confounders, or complex interactions. To address this, we introduce an analytically exact framework for the controlled behavioral evaluation of LLMs. We bridge human psychometrics with LLM mechanics by resolving gaps in design, measurement, and analysis. First, we replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects. Second, we eliminate Monte Carlo text sampling noise by operating directly on exact, token-level Probability Mass Functions (PMFs). Third, we derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically. We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.10503 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:08:50.317Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).