Metraipolis › archive › dc_95fcf57ab111
Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates — arXiv:2610.27745
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.27745
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-21T07:08:02.938Z
- SHA-256 of the archived text
d5e1168c19cd2ec556dbe8debb62b825e6f7c081db839bf142576a5c2cfba456- Stable URL of this page
https://metraipolis.com/documento/dc_95fcf57ab111
How to cite
Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates — arXiv:2610.27745 — arXiv (en), https://arxiv.org/abs/2610.27745 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_95fcf57ab111 (Metraipolis archive dc_95fcf57ab111, sha256 d5e1168c19cd2ec556dbe8debb62b825e6f7c081db839bf142576a5c2cfba456)
BibTeX
@misc{dc95fcf57ab111,
title = {Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates — arXiv:2610.27745},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.27745},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_95fcf57ab111; sha256 d5e1168c19cd2ec556dbe8debb62b825e6f7c081db839bf142576a5c2cfba456},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27745. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly used as judgment assistants and as stand-in participants in psychology experiments, yet little is known about how much their probability estimates move in response to a single new piece of information. We pair premise-hypothesis items from $\delta$-SNLI with one strengthener and one weakener each, elicit the prior and the posteriors in separate calls, and measure the directional movement s: the mean posterior change signed by the annotated direction, so that the prior cancels out. We vary whether the update is stated ("Here is something else you learned") or reported ("Someone said that"), and whether the model gives one probability or five weighted estimates. We test seven configurations: two local models (Qwen3-8B, OLMo-2-7B; 240 pairs) and five configurations run through agent command-line harnesses (120 pairs), with hypotheses and analysis rules fixed before the formal runs (not publicly registered). Under stated, direct elicitation all seven move in the annotated direction (s = 18-28 percentage points). Reporting lowers s in all seven point estimates (significantly in six, by 2-5 points) and raises the share of reverse movements in all seven: the effect is consistent but small. Five weighted estimates lower the share of responses at $\leq$5% or $\geq$95% in all seven, but in the two local models they also reduce s by 43% and 68%, whereas in the harnessed configurations the point estimates of the change lie between -3.7 and +0.5 points. An elicitation format that reduces extreme values thus need not preserve sensitivity to evidence; the two should be tested separately. Because $\delta$-NLI annotates only direction, we make no claim about how large an update should be. Code, data and the frozen plan: this https URL
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.27745 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.596Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).