Metraipolis › archive › dc_46aaa7679c40

Rethinking the Evaluation of LLM Fact-Checkers — arXiv:2610.30504

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.30504
License (as archived): CC0-1.0
Archived: 2026-10-25T08:35:47.488Z
SHA-256 of the archived text: 713d74db3f9c1f4c9b5432aabe944633d1623fb43cb2c71a25c7c3c13cdbc0d2
Stable URL of this page: https://metraipolis.com/documento/dc_46aaa7679c40

How to cite

Rethinking the Evaluation of LLM Fact-Checkers — arXiv:2610.30504 — arXiv (en), https://arxiv.org/abs/2610.30504 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_46aaa7679c40 (Metraipolis archive dc_46aaa7679c40, sha256 713d74db3f9c1f4c9b5432aabe944633d1623fb43cb2c71a25c7c3c13cdbc0d2)

BibTeX

@misc{dc46aaa7679c40,
  title = {Rethinking the Evaluation of LLM Fact-Checkers — arXiv:2610.30504},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30504},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_46aaa7679c40; sha256 713d74db3f9c1f4c9b5432aabe944633d1623fb43cb2c71a25c7c3c13cdbc0d2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30504. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design decisions: which questions the claims answer, under what conditions each claim is judged, and which sources count as evidence. No existing benchmark specifies all three. We introduce JSFactCheckBench, a Japanese benchmark that specifies all three, with 1,913 claims from LLM answers to questions about misleading information that circulated on Japanese social media, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Across six LLMs, all verify claims directly answering the question less accurately than supplementary claims, a gap that aggregate scores obscure. Scores also drop when models must locate the relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are added. These weaknesses are visible only because the benchmark records the conditions behind each label. A benchmark that leaves them implicit cannot reveal them.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.30504 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:27.752Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).