Metraipolis › archive › dc_1ccc0fcd0759

Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust — arXiv:2610.22610

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22610
License (as archived)
CC0-1.0
Archived
2026-10-19T09:21:18.757Z
SHA-256 of the archived text
440470c76860927bef04f30199a48543a79360d491dfaeb2cfe8f024466e2cd4
Stable URL of this page
https://metraipolis.com/documento/dc_1ccc0fcd0759

How to cite

Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust — arXiv:2610.22610 — arXiv (en), https://arxiv.org/abs/2610.22610 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1ccc0fcd0759 (Metraipolis archive dc_1ccc0fcd0759, sha256 440470c76860927bef04f30199a48543a79360d491dfaeb2cfe8f024466e2cd4)
BibTeX
@misc{dc1ccc0fcd0759,
  title = {Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust — arXiv:2610.22610},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22610},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1ccc0fcd0759; sha256 440470c76860927bef04f30199a48543a79360d491dfaeb2cfe8f024466e2cd4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22610. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language models (VLMs) are usually chosen from leaderboards, which give one ranking for every application. On questions about objects that a scene suggests but does not contain, however, the model with the lowest error on the POPE hallucination benchmark is not the one with the lowest error on such questions. For yes/no questions about object presence, we show that the best model for a target domain can be predicted without running any candidate on that domain, given 650 to 1,200 labeled target items that calibrate a detector. We measure each model's reliance on a cue, a feature such as scene context or text in the image, by editing 100 labeled source images so that the cue suggests the wrong answer, and comparing with a matched control edit. Reliance predicts which VLMs are misled by physical sticker attacks (Spearman .93 on nine models not used to design the score) and which hallucinate on unedited COCO photographs, beyond what their tendency to answer yes explains. Shift-Reliance Alignment (SRA) scores a model by its source error on the labeled source questions plus its reliance multiplied by the shift. The shift is the proportion of target questions on which the cue misleads, and a detector estimates it on the unlabeled target images. SRA selects the model with the lowest score. In pre-registered tests on new images where most questions mislead, it lowered the error of the selected model from 30.4% under selection by source error to 18.8% on Objects365 and OpenImages but, against its prediction, not on COCO. On CueDial, a benchmark we release that varies how often a cue misleads, selection by model confidence falls more than 40 percentage points behind the best model when overlaid text denies a visible object, and SRA stays within 3 points.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.22610 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:17:41.692Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).