Metraipolis › archive › dc_ca534be0aa36

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation — arXiv:2609.15561

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15561
License (as archived)
CC0-1.0
Archived
2026-10-15T23:16:11.305Z
SHA-256 of the archived text
21ad567f9a056ac49f32d1da0b2900845b2712663c5f015208bb76d49ed98d97
Stable URL of this page
https://metraipolis.com/documento/dc_ca534be0aa36

How to cite

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation — arXiv:2609.15561 — arXiv (en), https://arxiv.org/abs/2609.15561 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca534be0aa36 (Metraipolis archive dc_ca534be0aa36, sha256 21ad567f9a056ac49f32d1da0b2900845b2712663c5f015208bb76d49ed98d97)
BibTeX
@misc{dcca534be0aa36,
  title = {Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation — arXiv:2609.15561},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15561},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca534be0aa36; sha256 21ad567f9a056ac49f32d1da0b2900845b2712663c5f015208bb76d49ed98d97},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15561. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy? Despite the rise of factuality-based metrics, their sensitivity and reliability remain underexplored. This paper introduces a meta-evaluation framework that systematically tests these metrics using controlled corruptions of gold standard answers. Our method generates ranked outputs with known degrees of degradation to probe how metrics capture nuanced changes in truthfulness. Our experiments reveal that pipeline-based methods, such as the RAGAS's factual correctness metric, better track degradation than LLM-as-judge approaches. We also propose a new variant of the factual correctness metric that provides a competitive and cost-efficient.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.15561 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:51:09.950Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).