Metraipolis › archive › dc_989ace154512

Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms — arXiv:2610.39398

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.39398
License (as archived)
CC0-1.0
Archived
2026-10-31T06:17:00.484Z
SHA-256 of the archived text
105689fdaa55e638226da6e3c0d94f7132f54da05aec696c772d95469ba3e77c
Stable URL of this page
https://metraipolis.com/documento/dc_989ace154512

How to cite

Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms — arXiv:2610.39398 — arXiv (en), https://arxiv.org/abs/2610.39398 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_989ace154512 (Metraipolis archive dc_989ace154512, sha256 105689fdaa55e638226da6e3c0d94f7132f54da05aec696c772d95469ba3e77c)
BibTeX
@misc{dc989ace154512,
  title = {Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms — arXiv:2610.39398},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39398},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_989ace154512; sha256 105689fdaa55e638226da6e3c0d94f7132f54da05aec696c772d95469ba3e77c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39398. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Chatbots are increasingly consulted about public affairs, and their judgments of policies are useful only if they are robust. We measure this robustness in the setting of an ordinary user, through consumer chat interfaces. For 30 days, five chatbot configurations (Mistral, Gemini Flash, Claude Sonnet, and Claude Opus at two effort settings) were asked twice a day to score the long-term effect on the United States of 40 actions of the last four presidential administrations and of 15 hypothetical reforms, each with a confidence and a short comment. The judgments are robust: all configurations agree on the sign and the ranking of the administrations' mean scores, and two different configurations asked on the same day give opposite signs to the same action in about 4% of cases. Responses from different days differ more than same-day responses, but the difference grows slowly over the month, except where the service itself changed. We identify three such changes, which shifted the judgments while leaving little or no trace in the interface. Treating the comments as a retrieval task, we find them specific to each item and consistent across runs and chatbots, with lexical and embedding similarities agreeing. We release the dataset of 16,500 judgments.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2610.39398 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:23.358Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).