Metraipolis › archive › dc_6a96bbf7b14d
Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images — arXiv:2609.24190
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.24190
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T22:05:19.114Z
- SHA-256 of the archived text
faec688c8c45e644bca8df6e430d43e8cc777e7b5d051e62452d18f6cb252389- Stable URL of this page
https://metraipolis.com/documento/dc_6a96bbf7b14d
How to cite
Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images — arXiv:2609.24190 — arXiv (en), https://arxiv.org/abs/2609.24190 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6a96bbf7b14d (Metraipolis archive dc_6a96bbf7b14d, sha256 faec688c8c45e644bca8df6e430d43e8cc777e7b5d051e62452d18f6cb252389)
BibTeX
@misc{dc6a96bbf7b14d,
title = {Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images — arXiv:2609.24190},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.24190},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6a96bbf7b14d; sha256 faec688c8c45e644bca8df6e430d43e8cc777e7b5d051e62452d18f6cb252389},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24190. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Artificial intelligence (AI) has advanced at a rapid pace in recent years. Initially, breakthroughs in large language models caught widespread attention. However, recent generations of frontier AI models have adopted multimodal capabilities as a first class citizen, with vision capabilities being central to that. In this paper, we evaluate three recently released models on the task of diagnosing dermatological conditions from patient-submitted images. The models chosen are at the low to mid tier in terms of pricing and thus represent a floor on current AI capabilities, not a ceiling. We evaluate AI performance relative to a panel of three certified dermatologists, who grade each image, and we present four interesting findings. Firstly, depending on the metric, the tested AI models are either on par or slightly trail humans in terms of inter-clinician agreement. Secondly, we find that asking AI models for a confidence rating produces poorly calibrated answers, meaning use of confidence thresholds should not be relied upon in a clinical setting. Thirdly, the effect of providing additional patient metadata is strongly model-specific, with one of the three models degrading on every metric considered. Finally, model cost is not predictive of performance. The best-performing model we tested costs on average $0.0045 per case. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2609.24190 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T21:45:21.003Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).