Metraipolis › archive › dc_80fe4409e748
The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27953
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:47:20.950Z
- SHA-256 of the archived text
dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2- Stable URL of this page
https://metraipolis.com/documento/dc_80fe4409e748
How to cite
The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953 — arXiv (en), https://arxiv.org/abs/2608.27953 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80fe4409e748 (Metraipolis archive dc_80fe4409e748, sha256 dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2)
BibTeX
@misc{dc80fe4409e748,
title = {The Illusion of $textitWhat If$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27953},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80fe4409e748; sha256 dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27953. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present $\textbf{WhatIfBench}$, a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose $\textbf{PRISM}$, which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at $\href{ this https URL }{WhatIfBench}$.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27953 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.372Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).