Metraipolis › archive › dc_8632b239b36f
It Takes Two to Probe, Thousands to Retrieve: Putting Vision--Language Probes to the Retrieval Test — arXiv:2610.40124
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40124
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:17:24.685Z
- SHA-256 of the archived text
f229060bd28a4406b73092492820bd4ae6bcddad9d037783a5e7fcb3fc91dc7e- Stable URL of this page
https://metraipolis.com/documento/dc_8632b239b36f
How to cite
It Takes Two to Probe, Thousands to Retrieve: Putting Vision--Language Probes to the Retrieval Test — arXiv:2610.40124 — arXiv (en), https://arxiv.org/abs/2610.40124 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8632b239b36f (Metraipolis archive dc_8632b239b36f, sha256 f229060bd28a4406b73092492820bd4ae6bcddad9d037783a5e7fcb3fc91dc7e)
BibTeX
@misc{dc8632b239b36f,
title = {It Takes Two to Probe, Thousands to Retrieve: Putting Vision--Language Probes to the Retrieval Test — arXiv:2610.40124},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40124},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8632b239b36f; sha256 f229060bd28a4406b73092492820bd4ae6bcddad9d037783a5e7fcb3fc91dc7e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40124. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Vision--language models (VLMs) are tested for fine-grained understanding with image--text matching probes, a model must prefer the correct image or caption over handcrafted negative. In retrieval, the correct item must beat thousands of alternatives. We measure how these two evaluation methods relate by using two probing benchmarks, \svo{} and SugarCrepe, for retrieval tasks. For 100 image and caption queries we judge the top-10 retrievals from full dataset of six VLMs, with human annotators and an LLM assessor. We than compare how well models rank pool of items on the same queries, while changing only where the negatives comes from: the benchmark, a random draw, or the judged-incorrect retrievals of other models. On \svo{} the accuracy of retrieving a positive over handcrafted negatives is significantly higher than for retrieved ones (the correct item is preferred in 85\% against 77\% of the time). Using pool of ten retrieved negatives leads to same accuracy as when one uses pool of 200 to 430 random negatives. On SugarCrepe the order is reversed: its handcrafted negatives are harder than the retrieved ones. Incomplete relevance labels explain most of the low retrieval scores on \svo{}, where the top-ranked item is correct for 54--90\% of the queries under human labels against 6--21\% under the benchmark's labels. However, we detect no association between passing the probe for a query and retrieving a correct item first. The LLM assessor o3 agrees with individual annotators ($\kappa$ 0.57--0.61) slightly less than they agree with each other (0.64--0.68).
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.40124 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:23.417Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).