Metraipolis › archive › dc_81225fe2eae3

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering — arXiv:2608.11074

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11074
License (as archived)
CC0-1.0
Archived
2026-10-19T06:30:37.798Z
SHA-256 of the archived text
2f40f980b528a7a23f3ff68dc9f1abab9d48aa8437b9e88d6a615ee3bb799b4f
Stable URL of this page
https://metraipolis.com/documento/dc_81225fe2eae3

How to cite

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering — arXiv:2608.11074 — arXiv (en), https://arxiv.org/abs/2608.11074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_81225fe2eae3 (Metraipolis archive dc_81225fe2eae3, sha256 2f40f980b528a7a23f3ff68dc9f1abab9d48aa8437b9e88d6a615ee3bb799b4f)
BibTeX
@misc{dc81225fe2eae3,
  title = {CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering — arXiv:2608.11074},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11074},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_81225fe2eae3; sha256 2f40f980b528a7a23f3ff68dc9f1abab9d48aa8437b9e88d6a615ee3bb799b4f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-12 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.11074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:25:27.313Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).