Metraipolis › archive › dc_79326ca15125

Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers — arXiv:2611.06090

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06090
License (as archived)
CC0-1.0
Archived
2026-11-05T18:18:43.759Z
SHA-256 of the archived text
6a10e2e654d0ec581b4c1a5cbcd263f2110049c2fa909162d22cca52678992bc
Stable URL of this page
https://metraipolis.com/documento/dc_79326ca15125

How to cite

Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers — arXiv:2611.06090 — arXiv (en), https://arxiv.org/abs/2611.06090 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_79326ca15125 (Metraipolis archive dc_79326ca15125, sha256 6a10e2e654d0ec581b4c1a5cbcd263f2110049c2fa909162d22cca52678992bc)
BibTeX
@misc{dc79326ca15125,
  title = {Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers — arXiv:2611.06090},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06090},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_79326ca15125; sha256 6a10e2e654d0ec581b4c1a5cbcd263f2110049c2fa909162d22cca52678992bc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06090. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This paper presents an overview of \emph{SciClaimEval}, a pilot task at NTCIR-19 on verifying scientific claims against the tables and figures of the papers in which they appear. Our dataset pairs authentic claims from 180 papers in machine learning, natural language processing, and medicine with contrastive versions of their tables and figures, one supporting the claim and one minimally modified so that it does not. The task comprises two subtasks: deciding whether a piece of evidence supports a claim, and selecting which of two evidence versions does. Twenty-two teams registered, the highest number among NTCIR-19 tasks, and 14 submitted 177 runs. Most systems built on multimodal large language models, relying on prompting and multi-stage inference rather than task-specific fine-tuning. On the first subtask, the best systems reached 98.1\% pair accuracy, but all six top-ranked systems relied on the contrastive pairs; the best system that judged each claim-evidence pair on its own scored 24.3 points lower. Open-weight models, embedded in carefully designed pipelines, matched proprietary ones. Yet verifying a single claim against its evidence, the setting that matters in practice, remains an open problem.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.06090 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:13:03.943Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).