Metraipolis › archive › dc_b479fca539b7
A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents — arXiv:2609.32288
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32288
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:00:01.519Z
- SHA-256 of the archived text
88a2390dfe606d892622718a26e8147a4a385c39a934d92ce8e0800db3b6f7f4- Stable URL of this page
https://metraipolis.com/documento/dc_b479fca539b7
How to cite
A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents — arXiv:2609.32288 — arXiv (en), https://arxiv.org/abs/2609.32288 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b479fca539b7 (Metraipolis archive dc_b479fca539b7, sha256 88a2390dfe606d892622718a26e8147a4a385c39a934d92ce8e0800db3b6f7f4)
BibTeX
@misc{dcb479fca539b7,
title = {A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents — arXiv:2609.32288},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32288},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b479fca539b7; sha256 88a2390dfe606d892622718a26e8147a4a385c39a934d92ce8e0800db3b6f7f4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32288. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model's clean data performance degrade as the poison rate $\varepsilon$ grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase $\Delta$ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law $\Delta\approx C\varepsilon^{a}$ with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on $\varepsilon$ around a nondegenerate clean data optimum, $\Delta$ is generically quadratic in $\varepsilon$, so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by $q_\star$, and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is $\epsilon^{q_\star/(q_\star+2)}$, whereas taking the ample-data limit first gives $\epsilon^{2-2/q_\star}$, and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.32288 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.277Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).