Metraipolis › archive › dc_6f1ef03c2bc2
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior — arXiv:2609.39827
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39827
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:45:48.725Z
- SHA-256 of the archived text
a73f39efbe82c116e6ef33d570304fd315b65277eef769f24aba57cb63427282- Stable URL of this page
https://metraipolis.com/documento/dc_6f1ef03c2bc2
How to cite
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior — arXiv:2609.39827 — arXiv (en), https://arxiv.org/abs/2609.39827 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f1ef03c2bc2 (Metraipolis archive dc_6f1ef03c2bc2, sha256 a73f39efbe82c116e6ef33d570304fd315b65277eef769f24aba57cb63427282)
BibTeX
@misc{dc6f1ef03c2bc2,
title = {Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior — arXiv:2609.39827},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39827},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f1ef03c2bc2; sha256 a73f39efbe82c116e6ef33d570304fd315b65277eef769f24aba57cb63427282},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39827. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.39827 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:14.157Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).