Metraipolis › archive › dc_4e0ed3de0e45
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models — arXiv:2609.06704
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.06704
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:56:44.192Z
- SHA-256 of the archived text
6e1e1b3918cc41c4ce523644f6ebc2e4a56d98ed5e9fe0b66ffaa1e5a6cfaffd- Stable URL of this page
https://metraipolis.com/documento/dc_4e0ed3de0e45
How to cite
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models — arXiv:2609.06704 — arXiv (en), https://arxiv.org/abs/2609.06704 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e0ed3de0e45 (Metraipolis archive dc_4e0ed3de0e45, sha256 6e1e1b3918cc41c4ce523644f6ebc2e4a56d98ed5e9fe0b66ffaa1e5a6cfaffd)
BibTeX
@misc{dc4e0ed3de0e45,
title = {Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models — arXiv:2609.06704},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.06704},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e0ed3de0e45; sha256 6e1e1b3918cc41c4ce523644f6ebc2e4a56d98ed5e9fe0b66ffaa1e5a6cfaffd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06704. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Chain-of-thought (CoT) may often look plausible, yet it may not faithfully reflect the model's decision-making process. While methods for measuring the faithfulness of CoTs for textual inputs have been increasingly introduced, using these methods for visual inputs is not straightforward. In this work, we adapt the family of counterfactual methods for measuring CoT faithfulness, namely the Counterfactual Test (CT) and Correlational Counterfactual Test (CCT), to visual inputs, and call them vCT and vCCT, respectively. Using vCT and vCCT, we benchmark eight recent open-source Vision Language Models (VLMs) on two datasets. Our analysis shows that CoTs do not reliably track visual evidence that influences model predictions: they may omit the removed object even when its removal causes a large prediction shift, yet mention it when the shift is small. We further find that Predict-then-Explain explanations align more strongly with perturbation-induced probability shifts than pre-answer CoTs, while binary vCT scores are often nearly saturated. We also include a reconstruction control, in which images pass through the same editing pipeline without object removal, and find that the main object-removal intervention induces larger shifts than reconstruction alone. We construct and release Counter-SNLI-VE and Counter-A-OKVQA, two datasets of image pairs that differ by a single object. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.06704 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.280Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).