Metraipolis › archive › dc_54981f535bfe
Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study — arXiv:2609.37848
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37848
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:33:06.939Z
- SHA-256 of the archived text
25ac8b50d8714cc9d887d2a7056c3dc767a982dad9fdfe0d75ec2ba8d7a0e0a5- Stable URL of this page
https://metraipolis.com/documento/dc_54981f535bfe
How to cite
Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study — arXiv:2609.37848 — arXiv (en), https://arxiv.org/abs/2609.37848 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_54981f535bfe (Metraipolis archive dc_54981f535bfe, sha256 25ac8b50d8714cc9d887d2a7056c3dc767a982dad9fdfe0d75ec2ba8d7a0e0a5)
BibTeX
@misc{dc54981f535bfe,
title = {Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study — arXiv:2609.37848},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37848},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_54981f535bfe; sha256 25ac8b50d8714cc9d887d2a7056c3dc767a982dad9fdfe0d75ec2ba8d7a0e0a5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37848. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even when it depends strongly on how the benchmark was evaluated. We study this problem on the widely used Kermany pediatric chest radiograph dataset using nine image classifiers and a controlled evaluation protocol. Under the same protocol, the eight pretrained backbones differ by only 0.026 AUROC. In contrast, changing whether the backbone is frozen or fine-tuned changes AUROC by 0.044 on average, and changing the decision threshold changes balanced accuracy by 0.090 on average. The official test split is also measurably different from the training pool: a partition classifier distinguishes them at AUC 0.697, rising to 0.898 for normal radiographs. Most strikingly, a classifier using only file properties, with no image anatomy, reaches 0.992 balanced accuracy within the training pool but falls to 0.496 on the official test split. Validation-fitted thresholds and calibration also transfer imperfectly. These results show that a high benchmark score can support different conclusions when the split, training policy, threshold, metric, calibration, and uncertainty are not communicated with it. We end with a seven-item reporting recommendation in which each item is tied to an effect measured in the study Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.37848 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.580Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).