Metraipolis › archive › dc_0ac9f43ea8a0

Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining — arXiv:2609.25482

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25482
License (as archived)
CC0-1.0
Archived
2026-10-08T13:46:23.248Z
SHA-256 of the archived text
7b555f5201968a5a0867b875832fcc4d20fec211a0df3ab193d55c73ccbaab7a
Stable URL of this page
https://metraipolis.com/documento/dc_0ac9f43ea8a0

How to cite

Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining — arXiv:2609.25482 — arXiv (en), https://arxiv.org/abs/2609.25482 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ac9f43ea8a0 (Metraipolis archive dc_0ac9f43ea8a0, sha256 7b555f5201968a5a0867b875832fcc4d20fec211a0df3ab193d55c73ccbaab7a)
BibTeX
@misc{dc0ac9f43ea8a0,
  title = {Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining — arXiv:2609.25482},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25482},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ac9f43ea8a0; sha256 7b555f5201968a5a0867b875832fcc4d20fec211a0df3ab193d55c73ccbaab7a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25482. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) pretraining conventionally returns the raw final iterate. This couples two design choices: the learning-rate schedule that generates the parameter trajectory and the estimator that constructs the deployed model (e.g. the raw final iterate or a checkpoint average). A schedule that promotes optimization progress may differ from one that minimizes variation in the raw final iterate. Separating these choices creates an opportunity to maintain progress late in training while reducing variation in the returned model. To this end, we propose \emph{Terminal Shrinkage Averaging (TSA)}, which interpolates between the raw final iterate and the average of recent checkpoints to balance recent progress against terminal variation. We analyze how TSA changes the preferred terminal learning-rate schedule under a local quadratic approximation and test this interaction through a sequence of controlled NanoChat experiments. Finally, we demonstrate that the resulting gains transfer to depth-22 NanoChat, where the combined schedule and estimator improve validation quality. A qualifying time-to-GPT-2 run also finishes faster than the public baseline used in our experiments, providing preliminary evidence of benchmark acceleration.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.25482 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.231Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).