Metraipolis › archive › dc_0e718fd168bf
Beyond Binary Detection: Type-Aware Hallucination Analysis and Hierarchical Detection in Financial Question Answering — arXiv:2611.02258
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02258
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:43:45.603Z
- SHA-256 of the archived text
9aa1656ed31c196fb79c3a9fe6b5f742c7a9ee242d27ab260d938fb61b9b1bf9- Stable URL of this page
https://metraipolis.com/documento/dc_0e718fd168bf
How to cite
Beyond Binary Detection: Type-Aware Hallucination Analysis and Hierarchical Detection in Financial Question Answering — arXiv:2611.02258 — arXiv (en), https://arxiv.org/abs/2611.02258 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0e718fd168bf (Metraipolis archive dc_0e718fd168bf, sha256 9aa1656ed31c196fb79c3a9fe6b5f742c7a9ee242d27ab260d938fb61b9b1bf9)
BibTeX
@misc{dc0e718fd168bf,
title = {Beyond Binary Detection: Type-Aware Hallucination Analysis and Hierarchical Detection in Financial Question Answering — arXiv:2611.02258},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02258},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0e718fd168bf; sha256 9aa1656ed31c196fb79c3a9fe6b5f742c7a9ee242d27ab260d938fb61b9b1bf9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02258. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models are increasingly used to answer questions over financial filings, where an unfaithful answer can mislead investment, compliance, or client decisions. Hallucination detectors are a common safeguard, but they are usually evaluated with a single binary score that reveals neither which kinds of errors analysesed nor why. This paper studies hallucination detection on PHANTOM, a long-context benchmark built from SEC filings, at the level of hallucination type. We first analyze 15,443 paired hallucinated and faithful answers without any detector and find that hallucinated answers are more than twice as long as faithful ones, and that lexical grounding separates the two classes only because faithful answers are bimodal. We then generate hallucinations in nine financially motivated categories, confirm each with a separate verification pass, and diagnose an LLM-based detector on 4,086 usable examples. Generation success varies from 66.3% to 100% across categories. The detector reaches 80.2% binary accuracy but only 44.7% subtype accuracy, never assigns two of the nine categories, and misses between 5.3% and 46.8% of hallucinations depending on category; these misses are unrelated to input length but rise with how far an answer departs from the correct one. Guided by this diagnosis, we propose a two-stage hierarchical detector that first decides whether an answer is hallucinated and then assigns a subtype, raising binary accuracy to 95.6% and subtype accuracy to 56.1%. The results show that type-level evaluation exposes detector weaknesses that aggregate metrics conceal. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computational Finance (q-fin.CP). Pagina del lavoro: https://arxiv.org/abs/2611.02258 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:39:50.269Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).