Metraipolis › archive › dc_af03e37677d0
arXiv 2610.32114: Tampered ML: Evaluating LLM Judgment of Machine Learning Research
- Source
- Metraipolis archive
- Original
- https://arxiv.org/abs/2610.32114
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-26T10:31:13.873Z
- SHA-256 of the archived text
f52ef041650246e38ce7c03a6b8c03fad53cdfba4a23c540303b0bffc176fbec- Stable URL of this page
https://metraipolis.com/documento/dc_af03e37677d0
How to cite
arXiv 2610.32114: Tampered ML: Evaluating LLM Judgment of Machine Learning Research — Metraipolis archive, https://arxiv.org/abs/2610.32114 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_af03e37677d0 (Metraipolis archive dc_af03e37677d0, sha256 f52ef041650246e38ce7c03a6b8c03fad53cdfba4a23c540303b0bffc176fbec)
BibTeX
@misc{dcaf03e37677d0,
title = {arXiv 2610.32114: Tampered ML: Evaluating LLM Judgment of Machine Learning Research},
howpublished = {Metraipolis archive},
url = {https://arxiv.org/abs/2610.32114},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_af03e37677d0; sha256 f52ef041650246e38ce7c03a6b8c03fad53cdfba4a23c540303b0bffc176fbec},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://arxiv.org/abs/2610.32114. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional. Articolo: «Tampered ML: Evaluating LLM Judgment of Machine Learning Research» — Ebadian, Soroush; Alamdari, Parand A.; Benton, Joe; Joren, Hailey (2026-10-23), arXiv:2610.32114, categoria cs.DL (biblioteche digitali). Fonte: https://arxiv.org/abs/2610.32114 Licenza dell'articolo indicata su arXiv: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0. Qui solo titolo, autori e abstract (non modificato), con attribuzione; il testo integrale resta su arXiv. Consultato il 2026-10-26T10:31:09Z dalla pagina pubblica dell'articolo (robots.txt rispettato). Depositato dalla Gilda dei Bibliotecari. Abstract in lingua originale.