Metraipolis › archive › dc_f0072e439ca2

From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes — arXiv:2609.40148

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.40148
License (as archived)
CC0-1.0
Archived
2026-10-07T13:46:57.210Z
SHA-256 of the archived text
ce21d28a0eae8429c2bb478399dc8d3d2f1ec8823976b5f5dda1c00504ebea9a
Stable URL of this page
https://metraipolis.com/documento/dc_f0072e439ca2

How to cite

From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes — arXiv:2609.40148 — arXiv (en), https://arxiv.org/abs/2609.40148 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f0072e439ca2 (Metraipolis archive dc_f0072e439ca2, sha256 ce21d28a0eae8429c2bb478399dc8d3d2f1ec8823976b5f5dda1c00504ebea9a)
BibTeX
@misc{dcf0072e439ca2,
  title = {From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes — arXiv:2609.40148},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.40148},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f0072e439ca2; sha256 ce21d28a0eae8429c2bb478399dc8d3d2f1ec8823976b5f5dda1c00504ebea9a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40148. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time $T_t=\sum_{s<t}\eta_s$ controls optimization progress, while $r_t=B_t/\eta_t$ controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in $3+3(+2)$ propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched $B/\eta$ paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in $3+3(+2)$ map.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.40148 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T13:18:10.516Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).