Metraipolis › archive › dc_9ab7fcdee431
Before You Poll with LLMs: A Deliberative Diagnostic Framework — arXiv:2609.15849
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15849
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:54:07.695Z
- SHA-256 of the archived text
c4d2f9f9bd68a3a1d678bcb621036bf41a07c7360aee418c2820fd5b336e2ef1- Stable URL of this page
https://metraipolis.com/documento/dc_9ab7fcdee431
How to cite
Before You Poll with LLMs: A Deliberative Diagnostic Framework — arXiv:2609.15849 — arXiv (en), https://arxiv.org/abs/2609.15849 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9ab7fcdee431 (Metraipolis archive dc_9ab7fcdee431, sha256 c4d2f9f9bd68a3a1d678bcb621036bf41a07c7360aee418c2820fd5b336e2ef1)
BibTeX
@misc{dc9ab7fcdee431,
title = {Before You Poll with LLMs: A Deliberative Diagnostic Framework — arXiv:2609.15849},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15849},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9ab7fcdee431; sha256 c4d2f9f9bd68a3a1d678bcb621036bf41a07c7360aee418c2820fd5b336e2ef1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15849. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This is critical for silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether personas hold the right opinions -- a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this. We introduce the Deliberative Polling Diagnostic Framework, which compares human and LLM belief shifts after identical informational interventions. Grounded in deliberative polling, it surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we find that every model fails, each in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this signature self-sycophancy: conformity to the model's internal stereotype of the persona rather than reasoning from the information provided. Our framework offers a concrete protocol: run the deliberative diagnostic before trusting LLM personas to mimic revised beliefs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY). Pagina del lavoro: https://arxiv.org/abs/2609.15849 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.937Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).