Metraipolis › archive › dc_c88ac69ebec4

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones — arXiv:2610.22075

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22075
License (as archived)
CC0-1.0
Archived
2026-10-16T10:39:04.270Z
SHA-256 of the archived text
1febc8ebb7cc1a32d2bdd2ad56f19061750532020b71f16a6b93d1c6e0111749
Stable URL of this page
https://metraipolis.com/documento/dc_c88ac69ebec4

How to cite

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones — arXiv:2610.22075 — arXiv (en), https://arxiv.org/abs/2610.22075 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c88ac69ebec4 (Metraipolis archive dc_c88ac69ebec4, sha256 1febc8ebb7cc1a32d2bdd2ad56f19061750532020b71f16a6b93d1c6e0111749)
BibTeX
@misc{dcc88ac69ebec4,
  title = {Retrofitting Recurrent Vision-Language Models from Pretrained Backbones — arXiv:2610.22075},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22075},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c88ac69ebec4; sha256 1febc8ebb7cc1a32d2bdd2ad56f19061750532020b71f16a6b93d1c6e0111749},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22075. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.22075 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:52:02.715Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).