Metraipolis › archive › dc_ef1c80ee6777
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination — arXiv:2608.07341
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.07341
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T09:47:01.009Z
- SHA-256 of the archived text
0b66b6fbb91c1c4f5cce59d27383be028efbb04702569cb5006674c4eef1de0d- Stable URL of this page
https://metraipolis.com/documento/dc_ef1c80ee6777
How to cite
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination — arXiv:2608.07341 — arXiv (en), https://arxiv.org/abs/2608.07341 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef1c80ee6777 (Metraipolis archive dc_ef1c80ee6777, sha256 0b66b6fbb91c1c4f5cce59d27383be028efbb04702569cb5006674c4eef1de0d)
BibTeX
@misc{dcef1c80ee6777,
title = {Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination — arXiv:2608.07341},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.07341},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef1c80ee6777; sha256 0b66b6fbb91c1c4f5cce59d27383be028efbb04702569cb5006674c4eef1de0d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07341. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.07341 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.245Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).