Metraipolis › archive › dc_ca75f487d19f
From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2 — arXiv:2610.41732
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41732
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T07:34:55.545Z
- SHA-256 of the archived text
2bff83003bde376bded887f4d52f5f72ae2832c1cfc6e72f6fc354f41d146015- Stable URL of this page
https://metraipolis.com/documento/dc_ca75f487d19f
How to cite
From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2 — arXiv:2610.41732 — arXiv (en), https://arxiv.org/abs/2610.41732 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca75f487d19f (Metraipolis archive dc_ca75f487d19f, sha256 2bff83003bde376bded887f4d52f5f72ae2832c1cfc6e72f6fc354f41d146015)
BibTeX
@misc{dcca75f487d19f,
title = {From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2 — arXiv:2610.41732},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41732},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca75f487d19f; sha256 2bff83003bde376bded887f4d52f5f72ae2832c1cfc6e72f6fc354f41d146015},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41732. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Whether a language model was trained on a document matters for copyright, privacy, and benchmark contamination. Yet most membership-inference (MI) evaluations on language models collect members and non-members after the fact, so success may reflect distribution shift and failure is inconclusive. We re-ran OLMo 2 1B's official 50-billion-token mid-training stage, replacing selected training chunks with documents from 7,814 pairs matched on domain and difficulty. For each pair, a coin flip determined which document was inserted and which was held out. The run released by the Allen Institute for AI (AI2) from the same seed saw identical data everywhere else and serves as the counterfactual. One exposure raises a previously unseen document's likelihood on average but rarely identifies it. Using the counterfactual as reference, the reference ratio has an area under the ROC curve (AUC) of 0.767 but, with one threshold across domains, catches 3.8% of members at a 1% false-positive rate (FPR). In contrast, 13 matched pairs give a paired test 80% power at the 5% level to detect training on a collection, given same-domain non-members and a close reference. A reference trained with another seed recovers 92% of the counterfactual's AUC advantage over chance, whereas reference-free attacks stay within 0.07 of chance when pooled over domains. The full official post-training recipe, re-run on both models, leaves 89% of the average effect. Even with a closely matched reference, substantial document-level variability limits individual membership inference. We will release the checkpoints, per-token log-probabilities, and the evaluation documents; the code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Cryptography and Security (cs.CR) ; Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.41732 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T07:27:12.668Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).