Metraipolis › archive › dc_5183ca93edff

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models — arXiv:2609.35860

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.35860
License (as archived): CC0-1.0
Archived: 2026-10-07T16:24:52.948Z
SHA-256 of the archived text: f65152d3f0347366779a1eb2831a59250970b72997f0767f865a644b91893a6f
Stable URL of this page: https://metraipolis.com/documento/dc_5183ca93edff

How to cite

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models — arXiv:2609.35860 — arXiv (en), https://arxiv.org/abs/2609.35860 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5183ca93edff (Metraipolis archive dc_5183ca93edff, sha256 f65152d3f0347366779a1eb2831a59250970b72997f0767f865a644b91893a6f)

BibTeX

@misc{dc5183ca93edff,
  title = {The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models — arXiv:2609.35860},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35860},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5183ca93edff; sha256 f65152d3f0347366779a1eb2831a59250970b72997f0767f865a644b91893a6f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35860. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sampling based consistency is widely used for hallucination detection, yet aggregate performance can conceal systematic differences in which errors are detectable. This work studies that heterogeneity across four language models and three factual question answering datasets. Partitioning hallucinations by answer agreement reveals high agreement (Ghost) and low agreement (Flickering) regimes with an apparent detectability gap of $0.35$ to $0.46$ AUC. Because the statistics used to define the regimes and measure this gap are strongly coupled ($|\rho|\approx0.94$ to $1.00$), the raw result is treated as a property of agreement based detection rather than independent evidence. After freezing regime assignments, lexical and semantic response dispersion preserve the asymmetry, with bootstrap $95\%$ intervals excluding zero in all $12$ model and dataset settings. A stricter test using individual diffusion trajectories and no cross seed information preserves the asymmetry across all three LLaDA datasets ($p<0.005$) and directionally across all three Dream datasets, with one reaching significance. The hard regime varies substantially in prevalence across models ($16\%$ to $77\%$), and matched prompts frequently change regimes between models. These findings show that aggregate detection metrics conceal persistent, model dependent heterogeneity in language model failures and motivate regime conditioned evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35860 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.727Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).