Metraipolis › archive › dc_a47d239ac9fd
The Margin Knows the Background: Uneven Abstention in Frozen Vision-Language Models — arXiv:2611.00026
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00026
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:39:45.144Z
- SHA-256 of the archived text
5a7a61e281e384956fa613012df0073aecf32f8865a16b5f0839f69f6945ecda- Stable URL of this page
https://metraipolis.com/documento/dc_a47d239ac9fd
How to cite
The Margin Knows the Background: Uneven Abstention in Frozen Vision-Language Models — arXiv:2611.00026 — arXiv (en), https://arxiv.org/abs/2611.00026 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a47d239ac9fd (Metraipolis archive dc_a47d239ac9fd, sha256 5a7a61e281e384956fa613012df0073aecf32f8865a16b5f0839f69f6945ecda)
BibTeX
@misc{dca47d239ac9fd,
title = {The Margin Knows the Background: Uneven Abstention in Frozen Vision-Language Models — arXiv:2611.00026},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00026},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a47d239ac9fd; sha256 5a7a61e281e384956fa613012df0073aecf32f8865a16b5f0839f69f6945ecda},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00026. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Selective prediction allows a model to abstain on uncertain inputs, but a single confidence threshold can disproportionately reject examples from particular groups. We study how confidence scores produce this imbalance in frozen vision-language models (OpenAI CLIP, LAION CLIP, SigLIP). We express a family of scores, including the margin and maximum logit, through the average and the difference of the two highest class logits, and ask which carries group information. We show that equal group coverage at every threshold is equivalent to independence between the score and group membership. A randomized margin baseline tests whether a reduction in coverage disparity beats what adding noise achieves. With score choices fixed before testing, the effect of choice depends on the dataset. On Waterbirds, the margin creates coverage gaps 2.5 to 5.5 times larger than maximum logit at 80% coverage, its group signal is in the difference component, and paired background swaps show the background moves it, although maximum logit increases overall error. On NICO++, maximum logit instead increases worst-context error by 5.4 to 6.6 points and offers no improvement over the randomized baseline for OpenAI CLIP. Across both datasets, how far a score pushes a group down its ranking tracks the coverage gap (Spearman 0.88, 0.92) better than plain group identifiability (0.61, 0.65). These results explain why a score that abstains more evenly on one dataset may be less effective on another and motivate judging group coverage and selective risk together against randomized baselines. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2611.00026 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:06:37.353Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).