Metraipolis › archive › dc_1a1afefbc79f
Sequential Pretraining Favors Large Models — arXiv:2610.09611
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09611
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:56:46.355Z
- SHA-256 of the archived text
c0643b510233935c6ff2141dd3d3315ecccefdaeca1b9a9c3f8e90ef0e886360- Stable URL of this page
https://metraipolis.com/documento/dc_1a1afefbc79f
How to cite
Sequential Pretraining Favors Large Models — arXiv:2610.09611 — arXiv (en), https://arxiv.org/abs/2610.09611 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1a1afefbc79f (Metraipolis archive dc_1a1afefbc79f, sha256 c0643b510233935c6ff2141dd3d3315ecccefdaeca1b9a9c3f8e90ef0e886360)
BibTeX
@misc{dc1a1afefbc79f,
title = {Sequential Pretraining Favors Large Models — arXiv:2610.09611},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09611},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1a1afefbc79f; sha256 c0643b510233935c6ff2141dd3d3315ecccefdaeca1b9a9c3f8e90ef0e886360},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09611. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.09611 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:50:37.701Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).