Metraipolis › archive › dc_43743ef13618
Relative Generalization Invariance of LLM Pretraining — arXiv:2609.33016
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33016
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:00:51.799Z
- SHA-256 of the archived text
85ec524b806713db07ad1446dc04935020222c9a53f589c7f46f8569e2f19283- Stable URL of this page
https://metraipolis.com/documento/dc_43743ef13618
How to cite
Relative Generalization Invariance of LLM Pretraining — arXiv:2609.33016 — arXiv (en), https://arxiv.org/abs/2609.33016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_43743ef13618 (Metraipolis archive dc_43743ef13618, sha256 85ec524b806713db07ad1446dc04935020222c9a53f589c7f46f8569e2f19283)
BibTeX
@misc{dc43743ef13618,
title = {Relative Generalization Invariance of LLM Pretraining — arXiv:2609.33016},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33016},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_43743ef13618; sha256 85ec524b806713db07ad1446dc04935020222c9a53f589c7f46f8569e2f19283},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.523Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).