Metraipolis › archive › dc_057ef0d6b623
Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks — arXiv:2609.34506
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34506
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:57:49.841Z
- SHA-256 of the archived text
de62ec739c92f4a431209d46ec6e2c1e59efd7f6dcc33e5481be5c272c33ef12- Stable URL of this page
https://metraipolis.com/documento/dc_057ef0d6b623
How to cite
Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks — arXiv:2609.34506 — arXiv (en), https://arxiv.org/abs/2609.34506 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_057ef0d6b623 (Metraipolis archive dc_057ef0d6b623, sha256 de62ec739c92f4a431209d46ec6e2c1e59efd7f6dcc33e5481be5c272c33ef12)
BibTeX
@misc{dc057ef0d6b623,
title = {Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks — arXiv:2609.34506},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34506},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_057ef0d6b623; sha256 de62ec739c92f4a431209d46ec6e2c1e59efd7f6dcc33e5481be5c272c33ef12},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34506. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Human-model alignment is critical for trustworthy AI-assisted decision-making systems. Yet, most work evaluates model predictions against single ground-truth labels, overlooking that humans themselves often disagree on labels, a signal of genuine ambiguity. We investigate whether models struggle on the same instances that humans find difficult. We measure this on two vision datasets (FER+ and CIFAR-10H) where multiple human annotations per image capture human disagreement patterns. We evaluate eight pretrained models across three architectures (ResNet, EfficientNet, MobileNetV3) in two parts: first, whether model uncertainty (softmax confidence, entropy) correlates with human disagreement, and second, whether predictive multiplicity measures (inter-model disagreement, Jensen-Shannon divergence) do. We find that it does not: alignment is weak in both dimensions. At the discrete label level, 50.4% of CIFAR-10H images and 33.5% of FER+ images receive multiple valid classifications from humans, while the models converge on only one. These instances represent a critical failure case where humans perceive ambiguity and would request expert review, yet models decide confidently. At the continuous score level, single-model uncertainty correlates weakly with human disagreement ($\rho = 0.24--0.55$), and predictive multiplicity provides only modest improvement. Widely-used uncertainty quantification methods do not reliably identify instances humans find ambiguous. Model uncertainty should not be treated as a trustworthy signal by default for decision-making in high-stakes scenarios. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.34506 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.566Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).