Metraipolis › archive › dc_f36aca0ea9df

Scaling Domain Data Repetition in LLM Pretraining — arXiv:2608.14071

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.14071
License (as archived): CC0-1.0
Archived: 2026-10-19T02:20:31.918Z
SHA-256 of the archived text: 604a937636f9826463fe48943c6d2578420354baea76ca600bca655a976f0e1b
Stable URL of this page: https://metraipolis.com/documento/dc_f36aca0ea9df

How to cite

Scaling Domain Data Repetition in LLM Pretraining — arXiv:2608.14071 — arXiv (en), https://arxiv.org/abs/2608.14071 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f36aca0ea9df (Metraipolis archive dc_f36aca0ea9df, sha256 604a937636f9826463fe48943c6d2578420354baea76ca600bca655a976f0e1b)

BibTeX

@misc{dcf36aca0ea9df,
  title = {Scaling Domain Data Repetition in LLM Pretraining — arXiv:2608.14071},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14071},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f36aca0ea9df; sha256 604a937636f9826463fe48943c6d2578420354baea76ca600bca655a976f0e1b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14071. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.14071 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.775Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).