Metraipolis › archive › dc_b8024f42fee2

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity — arXiv:2608.10723

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.10723
License (as archived): CC0-1.0
Archived: 2026-10-19T06:30:25.824Z
SHA-256 of the archived text: 728a26205f3b7e9c476c27ff6fbe451c4f40033ae961ab225fe4b71c7da9e2a3
Stable URL of this page: https://metraipolis.com/documento/dc_b8024f42fee2

How to cite

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity — arXiv:2608.10723 — arXiv (en), https://arxiv.org/abs/2608.10723 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b8024f42fee2 (Metraipolis archive dc_b8024f42fee2, sha256 728a26205f3b7e9c476c27ff6fbe451c4f40033ae961ab225fe4b71c7da9e2a3)

BibTeX

@misc{dcb8024f42fee2,
  title = {Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity — arXiv:2608.10723},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10723},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b8024f42fee2; sha256 728a26205f3b7e9c476c27ff6fbe451c4f40033ae961ab225fe4b71c7da9e2a3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10723. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision Transformers demonstrate remarkable global modeling capacity but often underperform in data-scarce regimes. Distilling convolutional inductive biases from a CNN teacher provides an effective remedy while leaving the deployed model unchanged. However, general-purpose feature distillation transfers little in this setting. In CNN-to-CNN distillation, pooling, flattening, and logit-space projections remove the spatial grid that encodes locality and translation equivariance. Unlike a convolutional student, a ViT cannot readily reconstruct this structure on its own. In this paper, we propose iBKD, a distillation framework that preserves the spatial grid throughout the entire transfer process. Its core module, the Inductive Bias Attention Module, aggregates features from all student layers onto the teacher's grid using learned weights. It then enhances structural cues through channel and deformable spatial attention and injects them via convolutional cross-attention operating directly between spatial grids rather than token sets. The module is used only during training, leaving the deployed model as an unmodified ViT with no inference overhead. Across seven Transformer backbones and six data-scarce benchmarks, iBKD consistently outperforms both locality-guidance methods and general knowledge distillation baselines, with its advantage increasing as the amount of training data decreases.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-12 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.10723 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:25:27.277Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).