Metraipolis › archive › dc_b09af758e606
Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality — arXiv:2609.27607
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.27607
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T09:12:51.537Z
- SHA-256 of the archived text
dcf45059d45f0b4a1c02d49974fbe1928913353a4ae3d88465aeb4a3316e6b04- Stable URL of this page
https://metraipolis.com/documento/dc_b09af758e606
How to cite
Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality — arXiv:2609.27607 — arXiv (en), https://arxiv.org/abs/2609.27607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b09af758e606 (Metraipolis archive dc_b09af758e606, sha256 dcf45059d45f0b4a1c02d49974fbe1928913353a4ae3d88465aeb4a3316e6b04)
BibTeX
@misc{dcb09af758e606,
title = {Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality — arXiv:2609.27607},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.27607},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b09af758e606; sha256 dcf45059d45f0b4a1c02d49974fbe1928913353a4ae3d88465aeb4a3316e6b04},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its presence. Measuring these factual differences is essential for evaluating report generators. We study Jev, a System One decision model, as a simple, low-cost judge of agreement with physician-written reference reports. Our evaluator checks whether each statement is supported by the other report and combines these judgments in both directions to capture unsupported claims and omissions. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert with expert error counts, outperforming an open natural language inference judge under matched decomposition and aggregation. One support question per statement retains similar expert agreement to seven while using 43-45% fewer judgment input tokens. At the documented API price, judgments cost under three cents per hundred report pairs, excluding local decomposition. In a separate controlled-error test, Jev detects false negation with an AUROC of 0.977. Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset. Finding-count and error-scope analyses show that benchmark agreement reflects report size and error definitions as well as medical error detection. These results support Jev as a practical judgment component for measuring factual differences in generated radiology reports and identify where more elaborate evaluation remains valuable. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.27607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T08:13:26.697Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).