Metraipolis › archive › dc_bf8ccac26296

Evaluating Evidence Sensitivity and Robustness of Hallucination Detectors in Retrieval- Augmented Generation (2026) · DOI 10.21203/rs.3.rs-11400108/v1

Source
Metraipolis archive
Original
https://doi.org/10.21203/rs.3.rs-11400108/v1
License (as archived)
CC-BY-4.0
Archived
2026-10-26T19:00:12.205Z
SHA-256 of the archived text
348775d169537d43c2977818d9f538c224824f50677a8aa3f78c0d100ea53e6f
Stable URL of this page
https://metraipolis.com/documento/dc_bf8ccac26296

How to cite

Evaluating Evidence Sensitivity and Robustness of Hallucination Detectors in Retrieval- Augmented Generation (2026) · DOI 10.21203/rs.3.rs-11400108/v1 — Metraipolis archive, https://doi.org/10.21203/rs.3.rs-11400108/v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_bf8ccac26296 (Metraipolis archive dc_bf8ccac26296, sha256 348775d169537d43c2977818d9f538c224824f50677a8aa3f78c0d100ea53e6f)
BibTeX
@misc{dcbf8ccac26296,
  title = {Evaluating Evidence Sensitivity and Robustness of Hallucination Detectors in Retrieval- Augmented Generation (2026) · DOI 10.21203/rs.3.rs-11400108/v1},
  howpublished = {Metraipolis archive},
  url = {https://doi.org/10.21203/rs.3.rs-11400108/v1},
  note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_bf8ccac26296; sha256 348775d169537d43c2977818d9f538c224824f50677a8aa3f78c0d100ea53e6f},
  year = {2026}
}

Archived text

Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.21203/rs.3.rs-11400108/v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content).
Titolo: Evaluating Evidence Sensitivity and Robustness of Hallucination Detectors in Retrieval- Augmented Generation
Autori: Ramana Kulanthaivelu
Pubblicato: 2026-10-26 · editore: Springer Science and Business Media LLC
DOI: https://doi.org/10.21203/rs.3.rs-11400108/v1
Licenza dell'articolo: CC-BY-4.0

Abstract (testo degli autori, CC-BY-4.0):
Abstract Retrieval-Augmented Generation (RAG) improves the accessibility of external knowledge for large language models, but generated responses can remain unsupported, contradictory, or otherwise hallucinatory. Conventional hallucination-detector evaluation largely emphasizes aggregate benchmark performance, which may not reveal whether detector decisions remain reliable when the evidence distribution changes. This study presents a controlled evaluation of hallucination detectors from an evidence-sensitivity and robustness perspective. Three detector configurations - TF-IDF with Logistic Regression, TF-IDF with Linear Support Vector Machine, and DeBERTa-v3-base were evaluated using RAGTruth, HaluBench, and TRIVIA+. In-domain evaluation on RAGTruth showed that DeBERTa achieved an AUROC of 0.8483 and Macro-F1 of 0.7616. However, performance declined substantially under distribution shift, with AUROC decreasing to 0.5348 on HaluBench and 0.5910 on TRIVIA+. Controlled evidence interventions showed substantial detector responsiveness, with mean Evidence Sensitivity Index values of 0.2641 for evidence removal and 0.3268 for cross-example evidence substitution. A context-complexity stress test further showed progressive degradation as irrelevant evidence increased from 0% to 100% within a fixed 512-token observation window, with AUROC decreasing from 0.8306 to 0.7325 and Expected Calibration Error increasing from 0.1375 to 0.4072. Input ablation demonstrated that both generated answers and retrieved evidence contribute to detection performance. These findings show that benchmark accuracy alone is insufficient to characterize hallucination-detector reliability and motivate evaluation frameworks that jointly assess generalization, evidence sensitivity, contextual robustness, and calibration.

Fonte: https://api.crossref.org/works/10.21203%2Frs.3.rs-11400108%2Fv1 — metadati Crossref, consultati il 2026-10-26. Interrogazione: «retrieval augmented generation».