Metraipolis › archive › dc_eb7ee44625a6

Procedural Core: A Compact Recurrent Initialization for Vision Transformers — arXiv:2609.37631

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37631
License (as archived)
CC0-1.0
Archived
2026-10-07T17:31:39.682Z
SHA-256 of the archived text
bd7339c47d0932797281160cd1c949b7f940825ee3a425adefd6f253b814f422
Stable URL of this page
https://metraipolis.com/documento/dc_eb7ee44625a6

How to cite

Procedural Core: A Compact Recurrent Initialization for Vision Transformers — arXiv:2609.37631 — arXiv (en), https://arxiv.org/abs/2609.37631 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb7ee44625a6 (Metraipolis archive dc_eb7ee44625a6, sha256 bd7339c47d0932797281160cd1c949b7f940825ee3a425adefd6f253b814f422)
BibTeX
@misc{dceb7ee44625a6,
  title = {Procedural Core: A Compact Recurrent Initialization for Vision Transformers — arXiv:2609.37631},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37631},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb7ee44625a6; sha256 bd7339c47d0932797281160cd1c949b7f940825ee3a425adefd6f253b814f422},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37631. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that must be repeated for every target model. We propose Procedural Core, an initialization strategy that captures this generic structure into a compact set of weights that can be reused across models. We train a minimal recurrent transformer on procedural data, then expand its weights to initialize transformers of arbitrary width and depth. The resulting initialization improves performance on image classification, self-supervised visual learning (DINO), and modeling natural language (FineWeb-Edu) and code (CodeParrot). For image classification, expanding a 1M-parameter core to initialize an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by 2.2 pp over standard random initialization. Our analysis identifies recurrence as essential for learning compact weights that transfer across models. In ViTs, we localize a key benefit in the suppression of high-norm tokens that produces substantial improvements in zero-shot segmentation (ImageNet-S mAP 32.3 to 42.9), object localization (VOC07 CorLoc 9.9 to 18.4), and depth estimation (NYUv2 RMSE 1.104 to 0.998). This demonstrates that transformers need not start from a blank slate, and can be initialized with generic capabilities at low cost with no domain- or task-specific data.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.37631 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.354Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).