Metraipolis › archive › dc_2dac9b398a2e
Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective — arXiv:2609.21888
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21888
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:12:53.593Z
- SHA-256 of the archived text
4685ef27a328859d4b7b68079483e7ce3a16d22a15847c9c57fafaaef1c9d1f0- Stable URL of this page
https://metraipolis.com/documento/dc_2dac9b398a2e
How to cite
Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective — arXiv:2609.21888 — arXiv (en), https://arxiv.org/abs/2609.21888 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2dac9b398a2e (Metraipolis archive dc_2dac9b398a2e, sha256 4685ef27a328859d4b7b68079483e7ce3a16d22a15847c9c57fafaaef1c9d1f0)
BibTeX
@misc{dc2dac9b398a2e,
title = {Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective — arXiv:2609.21888},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21888},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2dac9b398a2e; sha256 4685ef27a328859d4b7b68079483e7ce3a16d22a15847c9c57fafaaef1c9d1f0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21888. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalization. In the joint space of prediction loss and predictive entropy, a likelihood-only detector uses a horizontal boundary and can mistake predictable non-members for members. Motivated by this, we introduce an inclined boundary that evaluates prediction loss relative to predictive entropy. Our analysis shows that entropy correction can preserve the expected membership signal while reducing its variance, thereby improving standardized member--non-member separation. We further extend the mean--variance analysis to the more general setting with a nonzero mean entropy gap. Interestingly, this entropy-adjusted score admits a Helmholtz free-energy interpretation, leading to Energy Transfer Detection (ETD), which views pretraining data detection from a macroscopic residual free-energy transfer perspective. Extensive experiments show that ETD achieves the best average detection performance, improving average AUROC by up to 3.5\% and TPR@5\%FPR by up to 5.1\%, while remaining robust across diverse settings. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.21888 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T00:23:42.574Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).