Metraipolis › archive › dc_f2389a33eb43
Calibrated Yes/No Visual Decisions with Small Open Vision-Language Models: A Pre-Registered Benchmark of 12 Models, Plug-In Heads, and an Airline Baggage Case Study (2026) · DOI 10.20944/preprints202…
- Source
- Metraipolis archive
- Original
- https://doi.org/10.20944/preprints202610.0848.v1
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-18T06:00:59.374Z
- SHA-256 of the archived text
1aa06818eee2b95bbdd8cb6a8184f03b233c51e912bf96719ccfeabd2c96ec7b- Stable URL of this page
https://metraipolis.com/documento/dc_f2389a33eb43
How to cite
Calibrated Yes/No Visual Decisions with Small Open Vision-Language Models: A Pre-Registered Benchmark of 12 Models, Plug-In Heads, and an Airline Baggage Case Study (2026) · DOI 10.20944/preprints202… — Metraipolis archive, https://doi.org/10.20944/preprints202610.0848.v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_f2389a33eb43 (Metraipolis archive dc_f2389a33eb43, sha256 1aa06818eee2b95bbdd8cb6a8184f03b233c51e912bf96719ccfeabd2c96ec7b)
BibTeX
@misc{dcf2389a33eb43,
title = {Calibrated Yes/No Visual Decisions with Small Open Vision-Language Models: A Pre-Registered Benchmark of 12 Models, Plug-In Heads, and an Airline Baggage Case Study (2026) · DOI 10.20944/preprints202…},
howpublished = {Metraipolis archive},
url = {https://doi.org/10.20944/preprints202610.0848.v1},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_f2389a33eb43; sha256 1aa06818eee2b95bbdd8cb6a8184f03b233c51e912bf96719ccfeabd2c96ec7b},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.20944/preprints202610.0848.v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content).
Titolo: Calibrated Yes/No Visual Decisions with Small Open Vision-Language Models: A Pre-Registered Benchmark of 12 Models, Plug-In Heads, and an Airline Baggage Case Study
Autori: Hasan Tatar
Pubblicato: 2026-10-10 · editore: MDPI AG
DOI: https://doi.org/10.20944/preprints202610.0848.v1
Licenza dell'articolo: CC-BY-4.0
Abstract (testo degli autori, CC-BY-4.0):
Many industrial vision questions are binary ("is this suitcase damaged?") and must be answered on premises, with a confidence that can be trusted and an option to defer to a human. We study how far small open vision-language models (VLMs) get on such questions and what it takes to make them reliable. First, a pre-registered benchmark of 12 open VLMs (0.8B-12B parameters; 1,807 yes/no items in four question families; test split analysed once) on an NVIDIA Jetson AGX Orin finds flat discrimination among the Qwen-based models of 2B parameters and above (pooled test AUROC 0.930-0.940), no gain from models post-trained for decisions over their own base models, and calibration that varies far more across models than ranking does: raw expected calibration error ranges from 0.014 to 0.122, and one shared Platt step brings every model to 0.013-0.037. Qwen3-VL-2B is the fastest model (0.45 s per image, 4.4 GB), and no model was significantly better than it in any tested comparison. Second, we attach per-task heads, L2-regularised logistic regressions on the frozen model's last-layer hidden state, and abstain below a confidence threshold set by the application. On test photos from sources unseen in training, a suitcase-damage head raises AUROC from 0.925 (calibrated model answer) to 0.962, paired 95% CI of the gain [+0.018, +0.058]; at a 0.90 threshold it answers 71.7% of the photos with 96.6% accuracy, against 61.1% for the calibrated answer. A baggage-presence head does not improve AUROC. Third, a case study runs the calibrated model on an 8 GB laptop: on 148 test photos it decides 49% automatically at 97.3% accuracy and leaves the rest to staff, at no API cost. Limitations: internet photos only, four question families, and a source confound in one damage test set.
Fonte: https://api.crossref.org/works/10.20944%2Fpreprints202610.0848.v1 — metadati Crossref, consultati il 2026-10-18. Interrogazione: «small language models».