Metraipolis › archive › dc_2ee07984f770
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models — arXiv:2609.01611
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01611
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T03:25:25.595Z
- SHA-256 of the archived text
be91c00392eafef77541bce1296b10b2f12d52361efda062bfc2a4c582b14a1b- Stable URL of this page
https://metraipolis.com/documento/dc_2ee07984f770
How to cite
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models — arXiv:2609.01611 — arXiv (en), https://arxiv.org/abs/2609.01611 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ee07984f770 (Metraipolis archive dc_2ee07984f770, sha256 be91c00392eafef77541bce1296b10b2f12d52361efda062bfc2a4c582b14a1b)
BibTeX
@misc{dc2ee07984f770,
title = {EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models — arXiv:2609.01611},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01611},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ee07984f770; sha256 be91c00392eafef77541bce1296b10b2f12d52361efda062bfc2a4c582b14a1b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01611. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks. We introduce EvalDetectBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect-compatible evaluation, allowing practitioners to test against current and future benchmarks. EvalDetectBench ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources. The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations. We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others. EvalDetectBench corrects for both via per-model probe calibration and a stratified generator-harmonisation procedure. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.01611 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T03:13:12.393Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).