Metraipolis › archive › dc_2509919bfda7

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344

Source
arXiv
Original
https://arxiv.org/abs/2610.02344
License (as archived)
CC0-1.0
Archived
2026-10-05T23:08:45.413Z
SHA-256 of the archived text
23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8
Stable URL of this page
https://metraipolis.com/documento/dc_2509919bfda7

How to cite

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344 — arXiv, https://arxiv.org/abs/2610.02344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2509919bfda7 (Metraipolis archive dc_2509919bfda7, sha256 23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8)
BibTeX
@misc{dc2509919bfda7,
  title = {Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures — arXiv:2610.02344},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02344},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2509919bfda7; sha256 23b0b476c3d6b3aa645039e3d29bd2183a31a81816929d5c76537db2437d24b8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Joint-Embedding Predictive Architectures (JEPAs) are prone to representation collapse, typically mitigated through empirical heuristics. We develop an early-training stability theory that unifies these heuristics. Linearising the coupled JEPA gradient flow around the trivial fixed point reveals two competing effects: a driving force ($\gamma$) and a decay effect ($\sigma$). Under approximate spectral decoupling, a per-mode stability ratio $\mu_i = \gamma_i / \sigma_i$ factorises into independent data-side and predictor-side terms and the count of unstable modes tracks the rank of representations that can emerge. The framework predicts a phase boundary, which we confirm empirically across more than 800 Tabular-JEPA configurations. It also unifies predictor scaling, masking ratio, and EMA as distinct mechanisms for shifting $\mu$. Guided by this analysis, we introduce ResidualPred, a transformer predictor whose attention is biased toward the identity at initialisation; it improves both effective rank and downstream accuracy on tabular benchmarks and in I-JEPA pretraining on CIFAR-10, CIFAR-100, STL-10, and ImageNet. Our framework connects empirical collapse-avoidance heuristics to an explicit dynamical picture, yielding theory-driven stabilizers. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.02344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T23:01:38.564Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).