Metraipolis › archive › dc_5dce5bd3ceb5

HumanSightBench: A Benchmark for Human Visual Perception in MLLMs — arXiv:2611.12305

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.12305
License (as archived)
CC0-1.0
Archived
2026-11-10T18:44:02.439Z
SHA-256 of the archived text
4462a89d95da87c9cee6885e6058d8bc3dc90c70b3be525069aa3f52f3a7479a
Stable URL of this page
https://metraipolis.com/documento/dc_5dce5bd3ceb5

How to cite

HumanSightBench: A Benchmark for Human Visual Perception in MLLMs — arXiv:2611.12305 — arXiv (en), https://arxiv.org/abs/2611.12305 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5dce5bd3ceb5 (Metraipolis archive dc_5dce5bd3ceb5, sha256 4462a89d95da87c9cee6885e6058d8bc3dc90c70b3be525069aa3f52f3a7479a)
BibTeX
@misc{dc5dce5bd3ceb5,
  title = {HumanSightBench: A Benchmark for Human Visual Perception in MLLMs — arXiv:2611.12305},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.12305},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5dce5bd3ceb5; sha256 4462a89d95da87c9cee6885e6058d8bc3dc90c70b3be525069aa3f52f3a7479a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.12305. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.12305 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.688Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).