Metraipolis › archive › dc_8aef79ec7e5a

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection — arXiv:2608.12652

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12652
License (as archived)
CC0-1.0
Archived
2026-10-19T04:24:28.003Z
SHA-256 of the archived text
6df09923f76560e328a6f1d9573e30eace5e1113a6d8c99c6bf1131b3be2c34e
Stable URL of this page
https://metraipolis.com/documento/dc_8aef79ec7e5a

How to cite

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection — arXiv:2608.12652 — arXiv (en), https://arxiv.org/abs/2608.12652 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8aef79ec7e5a (Metraipolis archive dc_8aef79ec7e5a, sha256 6df09923f76560e328a6f1d9573e30eace5e1113a6d8c99c6bf1131b3be2c34e)
BibTeX
@misc{dc8aef79ec7e5a,
  title = {Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection — arXiv:2608.12652},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12652},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8aef79ec7e5a; sha256 6df09923f76560e328a6f1d9573e30eace5e1113a6d8c99c6bf1131b3be2c34e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12652. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at release. A recent alternative reads it off a linear probe on internal activations. We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against. The protocol reports a zero-sum contrast on the depth profile of probe accuracy, recentred on a level-matched placebo baseline, tested against a label-permutation null, with the reference set twice the size of the suspect set. Each choice replaces a simpler alternative we rejected on measurement. Reporting the level of excess separability rather than its shape makes the false positive rate track the size of the analyst's own control set, 0.03 to 0.99 under a true null. Contrasting against a flat depth profile rejects a true null 0.72 of the time when surface decodability rises with depth, and loses all power when it falls. On real transformers the protocol fails a test the simulations did not pose. The recentring subtracts an estimate, and the permutation null holds it fixed. Re-estimated across split seeds on four audits of contaminated checkpoints, its standard deviation is 1.30 to 1.56 times the null's own in every arm: what is subtracted to remove a bias is more variable than what it corrects. The one nominally significant result, p = 0.0075, becomes 0.0745 once that variance is propagated, and no verdict is issued. The simulations missed this because their surface key is the covariate driving item variation; on real text it is a proxy, and degrading key quality in simulation reproduces it. We add a companion measurement and a widened null. No arm shows contamination.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.12652 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:17:42.377Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).