Metraipolis › archive › dc_65bfd7c0f178

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges — arXiv:2610.38957

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.38957
License (as archived)
CC0-1.0
Archived
2026-11-01T02:41:47.630Z
SHA-256 of the archived text
8a6918a6add2453544b3d58134815650d354acc3e19ff6cdc71ca3c54a75eba7
Stable URL of this page
https://metraipolis.com/documento/dc_65bfd7c0f178

How to cite

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges — arXiv:2610.38957 — arXiv (en), https://arxiv.org/abs/2610.38957 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_65bfd7c0f178 (Metraipolis archive dc_65bfd7c0f178, sha256 8a6918a6add2453544b3d58134815650d354acc3e19ff6cdc71ca3c54a75eba7)
BibTeX
@misc{dc65bfd7c0f178,
  title = {What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges — arXiv:2610.38957},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.38957},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_65bfd7c0f178; sha256 8a6918a6add2453544b3d58134815650d354acc3e19ff6cdc71ca3c54a75eba7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38957. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent advances in Multimodal Large Language Models (MLLMs) have led to significant progress in visual reasoning capabilities. However, existing benchmarks curate image-question pairs pertaining to object categories with few instances, enabling trivial object referencing and limited visual ambiguity. In this work, we introduce MIMO-Bench, a benchmark for evaluating MLLMs in a Multi-Instance per Multi-Object categories (MIMO) setting. Each image-question pair in our benchmark pertains to instances belonging to object categories with multiple instances. The presence of such similar feature distractors causes hindrance in basic visual reasoning tasks, necessitating the referring expression to be composed of multiple visual subtasks to uniquely and unambiguously identify the target object. MIMO-Bench comprises 3,802 image-question pairs across attribute binding and numerical counting tasks. Despite impressive performances on existing benchmarks, both SOTA open-source (InternVL, QwenVL) and proprietary (Gemini, GPT-5) models exhibit substantially lower performance on MIMO-Bench, highlighting their limitations in the MIMO setting. The benchmark will be made publicly available.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.38957 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:16:43.905Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).