Metraipolis › archive › dc_ed15ff62c8b1

SABRE: Scalable and Automated Benchmarking of VLMs under Stress — arXiv:2608.07435

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.07435
License (as archived)
CC0-1.0
Archived
2026-10-19T09:47:14.264Z
SHA-256 of the archived text
1166bca4a1aa088e01792f9c3b1287f0fd4fced1bdd47a91f8fdd834dd951258
Stable URL of this page
https://metraipolis.com/documento/dc_ed15ff62c8b1

How to cite

SABRE: Scalable and Automated Benchmarking of VLMs under Stress — arXiv:2608.07435 — arXiv (en), https://arxiv.org/abs/2608.07435 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ed15ff62c8b1 (Metraipolis archive dc_ed15ff62c8b1, sha256 1166bca4a1aa088e01792f9c3b1287f0fd4fced1bdd47a91f8fdd834dd951258)
BibTeX
@misc{dced15ff62c8b1,
  title = {SABRE: Scalable and Automated Benchmarking of VLMs under Stress — arXiv:2608.07435},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.07435},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ed15ff62c8b1; sha256 1166bca4a1aa088e01792f9c3b1287f0fd4fced1bdd47a91f8fdd834dd951258},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07435. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.07435 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.276Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).