Metraipolis › archive › dc_ebc48ded9773
Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study — arXiv:2608.15483
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.15483
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T19:40:30.556Z
- SHA-256 of the archived text
7a4eb7363ff25fb42f46d0f0d7ac01e3cd6c38884d1ea475895dc6d05b815b00- Stable URL of this page
https://metraipolis.com/documento/dc_ebc48ded9773
How to cite
Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study — arXiv:2608.15483 — arXiv (en), https://arxiv.org/abs/2608.15483 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebc48ded9773 (Metraipolis archive dc_ebc48ded9773, sha256 7a4eb7363ff25fb42f46d0f0d7ac01e3cd6c38884d1ea475895dc6d05b815b00)
BibTeX
@misc{dcebc48ded9773,
title = {Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study — arXiv:2608.15483},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.15483},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebc48ded9773; sha256 7a4eb7363ff25fb42f46d0f0d7ac01e3cd6c38884d1ea475895dc6d05b815b00},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15483. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.15483 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T19:02:44.100Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).