Metraipolis › archive › dc_f970ac16829b

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training — arXiv:2608.19168

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19168
License (as archived)
CC0-1.0
Archived
2026-10-18T13:53:37.181Z
SHA-256 of the archived text
44930ea8850a340379988edfb9b39c9c207543670002946cf7651ae811ac9231
Stable URL of this page
https://metraipolis.com/documento/dc_f970ac16829b

How to cite

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training — arXiv:2608.19168 — arXiv (en), https://arxiv.org/abs/2608.19168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f970ac16829b (Metraipolis archive dc_f970ac16829b, sha256 44930ea8850a340379988edfb9b39c9c207543670002946cf7651ae811ac9231)
BibTeX
@misc{dcf970ac16829b,
  title = {Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training — arXiv:2608.19168},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19168},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f970ac16829b; sha256 44930ea8850a340379988edfb9b39c9c207543670002946cf7651ae811ac9231},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A single training example's contribution to a finished model is normally estimated rather than measured, because measuring it takes two expensive full pre-training runs that differ in one row of one batch. We ran that counterfactual 24 times at a small scale. We trained 32 GPT-2 models at 124M parameters from scratch on OpenWebText, over four conditions and eight seeds. At step 200 of 9,536, at peak learning rate, we replaced one row of a 256-row batch with a fixed context injection carrying a 194-token passage. The three injected conditions are: 1. fluent prose with a corpus-attested subject, 2. fluent prose with a fabricated subject matched to it within 0.14% on full-batch gradient delta, and 3. random keyboard characters. The fourth condition is an uninjected twin. The passage is learned from one exposure and then decays. Fifty steps after injection, the arm that saw a passage predicts it better than the arm that did not by 0.039 and 0.044 nats of cross-entropy on the passage, at eight of eight seeds with p < 0.0001. At the final step we do not detect that difference for either passage, at p = 0.25 and p = 0.71, against minimum detectable effects of 0.025 and 0.079 nats, nor between the two passages, at p = 0.54. Every geometric measure we report is taken after that decay. Our pre-registered contrast on interpolation loss barrier is +0.0068 with p = 0.509, against a minimum detectable effect of 0.032 barrier units. Held-out cross-entropy is -0.00044 with p = 0.310. Per-layer centered kernel alignment does not detectably separate any condition at any layer. Weight displacement reaches 44.1% of the seed-to-seed Euclidean distance and is 92% settled by the midpoint of training, while the barrier reaches 3.0% of the seed-to-seed barrier. Those two figures sit roughly 15 times apart, and that is a lower bound. The injection relocates the model within its basin without moving it out.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.19168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:50:58.969Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).