Metraipolis › archive › dc_d7dfed727113
Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails — arXiv:2609.05535
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05535
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T15:06:29.930Z
- SHA-256 of the archived text
48c319fee272167f4ade4c635fa4f3fe078e3073467d35e732ac77c7275c609c- Stable URL of this page
https://metraipolis.com/documento/dc_d7dfed727113
How to cite
Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails — arXiv:2609.05535 — arXiv (en), https://arxiv.org/abs/2609.05535 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d7dfed727113 (Metraipolis archive dc_d7dfed727113, sha256 48c319fee272167f4ade4c635fa4f3fe078e3073467d35e732ac77c7275c609c)
BibTeX
@misc{dcd7dfed727113,
title = {Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails — arXiv:2609.05535},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05535},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d7dfed727113; sha256 48c319fee272167f4ade4c635fa4f3fe078e3073467d35e732ac77c7275c609c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05535. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Verdict-only evaluation does not reveal whether a vision-language model (VLM) used the visual evidence that should support its decision. We study this problem in web-agent guardrails, where a VLM judges whether on-screen text conflicts with a user instruction. We introduce Mind2Web-Injection, a benchmark of 9,954 instruction-screenshot pairs with instruction-relative labels, pixel-exact evidence boxes, and matched image-side counterfactuals. Across six VLMs, two models with nearly identical average precision differ ninefold in Evidence-Aligned Detection (EAD), the fraction of attacks both detected and correctly localized. To test whether a verdict depends on the command cited as evidence, we replace the instruction with one that endorses that command. Qwen3-VL-32B, the strongest open-weight localizer, returns aligned in only 58.7% of cases, whereas GPT-5.6-luna does so in 99.9%. To diagnose these failures, we propose two training-free interventions. ReadGate improves grounding without changing verdicts, while CmdCompare tests whether explicit instruction-command comparison resolves instruction-side inconsistency. These results motivate reporting verdict correctness, evidence localization, and counterfactual responsiveness separately. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.05535 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T14:49:56.972Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).