Metraipolis › archive › dc_68b88da463ce
Zero-shot evaluation of open vision-language models as System 1 decision models on radiology tasks — arXiv:2611.01631
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01631
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:18:05.097Z
- SHA-256 of the archived text
fd11f372838b20235c1e1173f8b990f0f3a84b4707ca211809b22173a04c8792- Stable URL of this page
https://metraipolis.com/documento/dc_68b88da463ce
How to cite
Zero-shot evaluation of open vision-language models as System 1 decision models on radiology tasks — arXiv:2611.01631 — arXiv (en), https://arxiv.org/abs/2611.01631 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_68b88da463ce (Metraipolis archive dc_68b88da463ce, sha256 fd11f372838b20235c1e1173f8b990f0f3a84b4707ca211809b22173a04c8792)
BibTeX
@misc{dc68b88da463ce,
title = {Zero-shot evaluation of open vision-language models as System 1 decision models on radiology tasks — arXiv:2611.01631},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01631},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_68b88da463ce; sha256 fd11f372838b20235c1e1173f8b990f0f3a84b4707ca211809b22173a04c8792},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01631. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Purpose: We compared two multimodal System 1 decision models, Clef 27B and d1 3B, with a traditional LLM (Qwen 3.8-27B) on seven radiology tasks covering text, radiographs, CT and MRI. Methods: Three models were compared: Clef, Qwen3.8-27B and d1. Three tasks were interpretation of text and 4 were detection of finding in images. The tasks were: 1. finding presence of finding in CT-RATE chest CT reports; 2. Presence of finding in PadChest chest radiograph reports; 3. the same 18 findings on CT-RATE chest CT but 3D volumes; 4. MR safety status of devices from AccessGUDID device records; 5. line and tube position on RANZCR CLiP chest radiographs; 6. motion-artifact severity on MR-ART T1-weighted brain MRI and 7. sequence identification on BraTS 2021 brain MRI. The primary endpoint was AUROC. There was no fine-tuning of the models on the tasks. Results: All models completed all tests without a failed request. All retrieved findings from reports almost perfectly (AUROC >0.95). Clef ranked higher than its Qwen backbone on CT-RATE volumes and RANZCR lower on MR-ART. The two did not differ on BraTS, GUDID or PadChest, and differed by 0.002 on CT-RATE reports. On BraTS, Clef correctly identified 90 of 100 FLAIR volumes while Qwen never chose it. d1 ranked lower than Clef on six of seven tasks. No model performed well on motion severity. Clef was faster per request than Qwen on tasks with many questions per input. d1 used 3 to 9 times less energy per request than Clef. Conclusions: System 1 models can perform important tasks in radiology with high accuracy, at lower compute cost than LLMs used in the usual way. When an LLM is constrained to a single answer token, its cost and performance is similar to System 1 models unless multiple questions are asked. A 3B System 1 model was close to the 27B models on report text but not on images. No model performed well on motion severity. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.01631 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.612Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).