Metraipolis › archive › dc_80fe4409e748

The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27953
License (as archived)
CC0-1.0
Archived
2026-10-17T10:47:20.950Z
SHA-256 of the archived text
dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2
Stable URL of this page
https://metraipolis.com/documento/dc_80fe4409e748

How to cite

The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953 — arXiv (en), https://arxiv.org/abs/2608.27953 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80fe4409e748 (Metraipolis archive dc_80fe4409e748, sha256 dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2)
BibTeX
@misc{dc80fe4409e748,
  title = {The Illusion of $textitWhat If$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs — arXiv:2608.27953},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27953},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80fe4409e748; sha256 dd29cd9af8bb8b9b9f0141b722262caebeffb910826ec4b1303638701a0872a2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27953. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present $\textbf{WhatIfBench}$, a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose $\textbf{PRISM}$, which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at $\href{ this https URL }{WhatIfBench}$.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27953 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.372Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).