Metraipolis › archive › dc_bc7108243350

When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04697
License (as archived)
CC0-1.0
Archived
2026-11-04T18:36:47.087Z
SHA-256 of the archived text
b665e11d643ec00708f0064e273e95fbd3de1ce15818e3f5d09b24905476dede
Stable URL of this page
https://metraipolis.com/documento/dc_bc7108243350

How to cite

When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697 — arXiv (en), https://arxiv.org/abs/2611.04697 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc7108243350 (Metraipolis archive dc_bc7108243350, sha256 b665e11d643ec00708f0064e273e95fbd3de1ce15818e3f5d09b24905476dede)
BibTeX
@misc{dcbc7108243350,
  title = {When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection — arXiv:2611.04697},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04697},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc7108243350; sha256 b665e11d643ec00708f0064e273e95fbd3de1ce15818e3f5d09b24905476dede},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04697. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hallucinations in Large Language Models (LLMs) remain a major challenge, as they can produce misleading, factually incorrect, or unverifiable information, thereby undermining their trustworthiness and reliability. However, reliable ground-truth labels remain a key challenge in evaluating hallucination detectors. Recent studies often construct labels automatically by comparing model generations with reference answers using similarity metrics such as BLEURT. This implicitly treats BLEURT similarity as a proxy for hallucination, despite its English-centric supervision, unbounded score range, and design as a similarity metric rather than a factuality measure, making threshold-based ground-truth construction potentially unreliable. Through comprehensive experiments using 15 automatic supervision judges, seven hallucination detection baselines, two generator models, two QA datasets, and five languages, we demonstrate that detector performance varies substantially with the choice of supervision judge. Human validation reveals substantial differences in labeling reliability, with QGen-9B achieving the lowest overall misclassification rate (12.86%), while BLEURT and BLEURT-n exhibit rates of 47.23% and 47.32%, respectively. Our findings highlight the limitations of automatic similarity-based labeling and the need for more reliable judging methods that better capture semantic and factual correctness.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.04697 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.458Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).