Metraipolis › archive › dc_9d5ffbef419b

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities — arXiv:2609.15695

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15695
License (as archived)
CC0-1.0
Archived
2026-10-15T22:48:02.878Z
SHA-256 of the archived text
7f4ddc8b2306151d9072a52a051f19529b7339e72317946b450438ccdc30fe0e
Stable URL of this page
https://metraipolis.com/documento/dc_9d5ffbef419b

How to cite

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities — arXiv:2609.15695 — arXiv (en), https://arxiv.org/abs/2609.15695 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9d5ffbef419b (Metraipolis archive dc_9d5ffbef419b, sha256 7f4ddc8b2306151d9072a52a051f19529b7339e72317946b450438ccdc30fe0e)
BibTeX
@misc{dc9d5ffbef419b,
  title = {NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities — arXiv:2609.15695},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15695},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9d5ffbef419b; sha256 7f4ddc8b2306151d9072a52a051f19529b7339e72317946b450438ccdc30fe0e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15695. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users on Xiaohongshu, a mainstream Chinese image-sharing platform, continue to turn to other people for help with everyday visual questions. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 9 frontier VLMs, the highest short-answer accuracy is 52.8\%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0\%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.15695 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.342Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).