Metraipolis › archive › dc_af8d98102cbe

Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision — arXiv:2608.04879

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04879
License (as archived)
CC0-1.0
Archived
2026-10-31T11:35:25.243Z
SHA-256 of the archived text
d146557c4d82446cd22ced9d3838adc45a650803c7cfccec4e157626012d6c7a
Stable URL of this page
https://metraipolis.com/documento/dc_af8d98102cbe

How to cite

Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision — arXiv:2608.04879 — arXiv (en), https://arxiv.org/abs/2608.04879 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_af8d98102cbe (Metraipolis archive dc_af8d98102cbe, sha256 d146557c4d82446cd22ced9d3838adc45a650803c7cfccec4e157626012d6c7a)
BibTeX
@misc{dcaf8d98102cbe,
  title = {Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision — arXiv:2608.04879},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04879},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_af8d98102cbe; sha256 d146557c4d82446cd22ced9d3838adc45a650803c7cfccec4e157626012d6c7a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04879. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth---at matched FLOPs or at matched parameter memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy--parameter trade-off under memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_\theta(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy--robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.04879 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:24:48.800Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).