Metraipolis › archive › dc_a580686d0899

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text — arXiv:2609.40295

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.40295
License (as archived)
CC0-1.0
Archived
2026-10-07T13:14:11.375Z
SHA-256 of the archived text
a9582cc18bbde65bf9499624c9e93c4517756b9e1afad3f3e7df2ad95006f9fb
Stable URL of this page
https://metraipolis.com/documento/dc_a580686d0899

How to cite

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text — arXiv:2609.40295 — arXiv (en), https://arxiv.org/abs/2609.40295 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a580686d0899 (Metraipolis archive dc_a580686d0899, sha256 a9582cc18bbde65bf9499624c9e93c4517756b9e1afad3f3e7df2ad95006f9fb)
BibTeX
@misc{dca580686d0899,
  title = {How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text — arXiv:2609.40295},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.40295},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a580686d0899; sha256 a9582cc18bbde65bf9499624c9e93c4517756b9e1afad3f3e7df2ad95006f9fb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40295. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August. Unlike synthetic data or model-collapse setups, this *wild* AI text comes from many models, is written for human readers, and arrives unlabeled in pretraining corpora. How does AI text in the wild affect language model pretraining? To answer this question, we pretrain 800 language models, varying the ratio of added AI tokens to human tokens, and fit scaling laws to held-out losses on both human and AI-generated text. For data-starved models, adding AI tokens to pretraining data initially lowers loss on human text, but the benefit saturates as more are added and quickly *reverses* into harm. For models trained on high budgets of human text, AI tokens raise loss almost immediately, while the same number of fresh human tokens keeps lowering it. Scaling laws such as Hoffman et al. (2022) fail to predict this behavior. We propose a new scaling law with separate benefit and harm terms that allows the value of an AI token to change sign while also reducing to Chinchilla in the absence of AI text. When fit on smaller models, our scaling law predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios. We recommend filtering AI text when the target is human text, repeating human text before expanding the training dataset with AI-generated web text, and reporting validation loss on human and AI text separately AI text remains valuable when the target is AI text. We release WildAI, an 83B-token corpus with AI, topic, and format labels, all 800 models and code at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.40295 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:44:51.906Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).