Metraipolis › archive › dc_48ff06bcaee7

Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis — arXiv:2610.21638

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.21638
License (as archived): CC0-1.0
Archived: 2026-10-16T10:02:17.429Z
SHA-256 of the archived text: 87bf1a151e400e3e7a8f1996b0744054ea9051d2a64af5fc478fd34a9ce3ff64
Stable URL of this page: https://metraipolis.com/documento/dc_48ff06bcaee7

How to cite

Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis — arXiv:2610.21638 — arXiv (en), https://arxiv.org/abs/2610.21638 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48ff06bcaee7 (Metraipolis archive dc_48ff06bcaee7, sha256 87bf1a151e400e3e7a8f1996b0744054ea9051d2a64af5fc478fd34a9ce3ff64)

BibTeX

@misc{dc48ff06bcaee7,
  title = {Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis — arXiv:2610.21638},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21638},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48ff06bcaee7; sha256 87bf1a151e400e3e7a8f1996b0744054ea9051d2a64af5fc478fd34a9ce3ff64},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21638. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision Transformers deployed in medical imaging exploit spurious correlations, yet the internal mechanisms remain unidentified and impossible to fix. We apply mechanistic interpretability to locate one such mechanism in ViT-Base 16: a single attention head (L8H10) that functions as a dedicated annotation-reader on chest X-rays, attending to corner text rather than lung pathology in 62% of 200 COVID-19 images. Causal validation via inpainting confirms annotation content rather than luminance or edge density drives the behaviour, increasing lung attention 1.52 times upon removal in affected images ($p{<}0.0001$). The finding replicates across NIH ChestX-ray14 (79%), DeiT-Base (88%), and ViT-Large 16 (80%), establishing spurious annotation-attending heads as a consistent property of ViT-family architectures across two datasets and three architectures. Full fine-tuning does not suppress the behaviour and may entrench it: 98.4% of spurious images remain spurious and the spurious rate rises marginally, as the shortcut is reinforced rather than removed. Critically, single-head ablation produces $p{=}0.340$ and multi-head ablation of all three identified carriers produces $p{=}0.532$, meaning a causally verified annotation-reading circuit is completely invisible to output-level evaluation. Mechanistic auditing finds what behavioural testing cannot.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.21638 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:47.757Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).