Metraipolis › archive › dc_8d18735fb759
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis — arXiv:2609.00746
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00746
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:52:12.992Z
- SHA-256 of the archived text
99213dab350cbe256b38e798206ed85c040bc4165780d499e0e9d9683e3a1b0b- Stable URL of this page
https://metraipolis.com/documento/dc_8d18735fb759
How to cite
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis — arXiv:2609.00746 — arXiv (en), https://arxiv.org/abs/2609.00746 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d18735fb759 (Metraipolis archive dc_8d18735fb759, sha256 99213dab350cbe256b38e798206ed85c040bc4165780d499e0e9d9683e3a1b0b)
BibTeX
@misc{dc8d18735fb759,
title = {Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis — arXiv:2609.00746},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00746},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d18735fb759; sha256 99213dab350cbe256b38e798206ed85c040bc4165780d499e0e9d9683e3a1b0b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00746. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.00746 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:41:55.817Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).