Metraipolis › archive › dc_0b784e68211e

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers — arXiv:2609.35745

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35745
License (as archived)
CC0-1.0
Archived
2026-10-07T21:03:55.539Z
SHA-256 of the archived text
5cf6a8cf75409f144b0090af8602d8fb2794526c4fe92f5e7f22c975e6adb60c
Stable URL of this page
https://metraipolis.com/documento/dc_0b784e68211e

How to cite

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers — arXiv:2609.35745 — arXiv (en), https://arxiv.org/abs/2609.35745 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0b784e68211e (Metraipolis archive dc_0b784e68211e, sha256 5cf6a8cf75409f144b0090af8602d8fb2794526c4fe92f5e7f22c975e6adb60c)
BibTeX
@misc{dc0b784e68211e,
  title = {Copy the Same, Distill the Difference: Initializing Linear Vision Transformers — arXiv:2609.35745},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35745},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0b784e68211e; sha256 5cf6a8cf75409f144b0090af8602d8fb2794526c4fe92f5e7f22c975e6adb60c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35745. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.35745 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.439Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).