Metraipolis › archive › dc_34752fb5ae13

When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04697
License (as archived)
CC0-1.0
Archived
2026-11-04T21:40:09.306Z
SHA-256 of the archived text
2114c9b2319405be50addc2f2797656e7738be6aa5bcd2056ea5c21381e0e2d7
Stable URL of this page
https://metraipolis.com/documento/dc_34752fb5ae13

How to cite

When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697 — arXiv (en), https://arxiv.org/abs/2611.04697 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_34752fb5ae13 (Metraipolis archive dc_34752fb5ae13, sha256 2114c9b2319405be50addc2f2797656e7738be6aa5bcd2056ea5c21381e0e2d7)
BibTeX
@misc{dc34752fb5ae13,
  title = {When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04697},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_34752fb5ae13; sha256 2114c9b2319405be50addc2f2797656e7738be6aa5bcd2056ea5c21381e0e2d7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04697. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hallucinations in Large Language Models (LLMs) remain a major challenge, as they can produce misleading, factually incorrect, or unverifiable information, thereby undermining their trustworthiness and reliability. However, reliable ground-truth labels remain a key challenge in evaluating hallucination detectors. Recent studies often construct labels automatically by comparing model generations with reference answers using similarity metrics such as BLEURT. This implicitly treats BLEURT similarity as a proxy for hallucination, despite its English-centric supervision, unbounded score range, and design as a similarity metric rather than a factuality measure, making threshold-based ground-truth construction potentially unreliable. Through comprehensive experiments using 15 automatic supervision judges, seven hallucination detection baselines, two generator models, two QA datasets, and five languages, we demonstrate that detector performance varies substantially with the choice of supervision judge. Human validation reveals substantial differences in labeling reliability, with QGen-9B achieving the lowest overall misclassification rate (12.86%), while BLEURT and BLEURT-n exhibit rates of 47.23% and 47.32%, respectively. Our findings highlight the limitations of automatic similarity-based labeling and the need for more reliable judging methods that better capture semantic and factual correctness.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.04697 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.683Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).