Metraipolis › archive › dc_444ed06aed96

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining — arXiv:2610.05591

Source
arXiv
Original
https://arxiv.org/abs/2610.05591
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:22.692Z
SHA-256 of the archived text
83496f2635af5ebe8f23fd49e0a64e3303cb64c8579c1d639dda0598e9f8ec0d
Stable URL of this page
https://metraipolis.com/documento/dc_444ed06aed96

How to cite

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining — arXiv:2610.05591 — arXiv, https://arxiv.org/abs/2610.05591 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_444ed06aed96 (Metraipolis archive dc_444ed06aed96, sha256 83496f2635af5ebe8f23fd49e0a64e3303cb64c8579c1d639dda0598e9f8ec0d)
BibTeX
@misc{dc444ed06aed96,
  title = {What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining — arXiv:2610.05591},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05591},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_444ed06aed96; sha256 83496f2635af5ebe8f23fd49e0a64e3303cb64c8579c1d639dda0598e9f8ec0d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05591. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters. We answer them by pricing a repeated token against two references: one epoch on the same data, which gives its value, and fresh data at equal compute, which gives its cost. Against fresh data, the cost of repetition follows a single variable, the number of extra epochs divided by the unique tokens per parameter. Against the same data, a second epoch is worth nearly as much as a fresh one, and repeated tokens fall to half the value of fresh ones after a critical epoch count that grows with the training budget per parameter but hardly with model size. With unique data fixed, the predicted compute-optimal run grows model size and epochs together until loss stops improving, near the critical epoch count. The same variable accounts for the direction of size trends that appear to conflict: larger models tolerate fewer epochs when the corpus is fixed, from about 15 at 127M to 4 at 2B parameters, but not when unique data grow with the model. Counts alone do not determine loss: at identical counts, replaying shards consecutively raises loss by up to 0.46~bits per byte, concentrating repeats on fewer samples also raises it, lower-entropy sources degrade faster with repetition, and re-tokenizing repeats helps only under heavy repetition. These results offer an empirical guide to pretraining when unique data, rather than compute, are the binding constraint.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.05591 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.337Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).