Metraipolis › archive › dc_85eef4c36a58

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models — arXiv:2609.10830

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.10830
License (as archived)
CC0-1.0
Archived
2026-10-16T05:40:40.327Z
SHA-256 of the archived text
b8788f2a4c1cb8e3a1a4127047aeb7970639a0fff2a15cd9a545cc2b2a4ee81e
Stable URL of this page
https://metraipolis.com/documento/dc_85eef4c36a58

How to cite

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models — arXiv:2609.10830 — arXiv (en), https://arxiv.org/abs/2609.10830 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_85eef4c36a58 (Metraipolis archive dc_85eef4c36a58, sha256 b8788f2a4c1cb8e3a1a4127047aeb7970639a0fff2a15cd9a545cc2b2a4ee81e)
BibTeX
@misc{dc85eef4c36a58,
  title = {Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models — arXiv:2609.10830},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.10830},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_85eef4c36a58; sha256 b8788f2a4c1cb8e3a1a4127047aeb7970639a0fff2a15cd9a545cc2b2a4ee81e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.10830. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near -0.08, where -1 would be a perfect relation and 0 none. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same, within noise, whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC, on a scale where 0.5 is a coin flip and 1.0 is perfect separation. We release the sentence banks, counts, and code.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Cryptography and Security (cs.CR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.10830 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T05:23:56.980Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).