Metraipolis › archive › dc_2509919bfda7
Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02344
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T23:08:45.413Z
- SHA-256 of the archived text
23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8- Stable URL of this page
https://metraipolis.com/documento/dc_2509919bfda7
How to cite
Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344 — arXiv, https://arxiv.org/abs/2610.02344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2509919bfda7 (Metraipolis archive dc_2509919bfda7, sha256 23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8)
BibTeX
@misc{dc2509919bfda7,
title = {Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02344},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2509919bfda7; sha256 23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Joint-Embedding Predictive Architectures (JEPAs) are prone to representation collapse, typically mitigated through empirical heuristics. We develop an early-training stability theory that unifies these heuristics. Linearising the coupled JEPA gradient flow around the trivial fixed point reveals two competing effects: a driving force ($\gamma$) and a decay effect ($\sigma$). Under approximate spectral decoupling, a per-mode stability ratio $\mu_i = \gamma_i / \sigma_i$ factorises into independent data-side and predictor-side terms and the count of unstable modes tracks the rank of representations that can emerge. The framework predicts a phase boundary, which we confirm empirically across more than 800 Tabular-JEPA configurations. It also unifies predictor scaling, masking ratio, and EMA as distinct mechanisms for shifting $\mu$. Guided by this analysis, we introduce ResidualPred, a transformer predictor whose attention is biased toward the identity at initialisation; it improves both effective rank and downstream accuracy on tabular benchmarks and in I-JEPA pretraining on CIFAR-10, CIFAR-100, STL-10, and ImageNet. Our framework connects empirical collapse-avoidance heuristics to an explicit dynamical picture, yielding theory-driven stabilizers. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.02344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T23:01:38.564Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).