Metraipolis › archive › dc_0fd068957fbd

Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters — arXiv:2609.27581

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27581
License (as archived)
CC0-1.0
Archived
2026-10-08T10:05:39.427Z
SHA-256 of the archived text
b423e10874e26c1a2c8ceeea0b5037131c69775cc3b8ca57e0af446442e04a5e
Stable URL of this page
https://metraipolis.com/documento/dc_0fd068957fbd

How to cite

Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters — arXiv:2609.27581 — arXiv (en), https://arxiv.org/abs/2609.27581 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0fd068957fbd (Metraipolis archive dc_0fd068957fbd, sha256 b423e10874e26c1a2c8ceeea0b5037131c69775cc3b8ca57e0af446442e04a5e)
BibTeX
@misc{dc0fd068957fbd,
  title = {Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters — arXiv:2609.27581},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27581},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0fd068957fbd; sha256 b423e10874e26c1a2c8ceeea0b5037131c69775cc3b8ca57e0af446442e04a5e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27581. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on models between 59M and 1B parameters; the small-model regime N < 59M was never tested empirically by its authors. This regime matters for single-GPU training, interpretability research, educational experiments, and settings where larger models are infeasible on memory or cost grounds. We test whether Step Law transfers to small language models. We consider three outcomes: H1, the original coefficients work directly; H2, the power-law form holds but with different coefficients; and H3, a power law does not describe the optima in this regime. All experiments use a single nanoGPT/TinyStories pipeline with a 2048-token BPE vocabulary, AdamW, and a warmup-cosine schedule. The optimum for each (N, D) cell is extracted from the loss surface L(eta, B) via a local quadratic approximation in log-log coordinates over the smoothed training loss. The final dataset contains 29 unique (N, D) cells and 935 analysis-ready runs. The main refit uses 25 cells (815 runs) in the working range 4 <= D/N <= 600. On the pooled data we accept H2: the functional form is preserved, but the coefficients differ from the original. We obtain eta*(N, D) = 0.0985 N^(-0.508) D^(0.238) (R^2 = 0.834) and B*(D) = 3.6 x 10^(-4) D^(0.931) (R^2 = 0.950). Step Law's structural claim that B* is independent of N is reproduced (p = 0.87), but the growth of B* with D is nearly twice as steep as in the original work. Direct transfer of Step Law systematically overestimates the optimal learning rate: the median ratio eta_SL / eta* is approximately 4.0x, with a range of 2.4x to 6.6x.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.27581 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:11:49.686Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).