Metraipolis › archive › dc_97e8daf88178

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research — arXiv:2608.26753

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26753
License (as archived)
CC0-1.0
Archived
2026-10-17T14:28:06.182Z
SHA-256 of the archived text
0c39c1d9cfd47cd72ad93330585c34c6fe697a7ae3de143861a9f7663a6614bc
Stable URL of this page
https://metraipolis.com/documento/dc_97e8daf88178

How to cite

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research — arXiv:2608.26753 — arXiv (en), https://arxiv.org/abs/2608.26753 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_97e8daf88178 (Metraipolis archive dc_97e8daf88178, sha256 0c39c1d9cfd47cd72ad93330585c34c6fe697a7ae3de143861a9f7663a6614bc)
BibTeX
@misc{dc97e8daf88178,
  title = {Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research — arXiv:2608.26753},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26753},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_97e8daf88178; sha256 0c39c1d9cfd47cd72ad93330585c34c6fe697a7ae3de143861a9f7663a6614bc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26753. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper's claims, and provide evidence supporting those claims. We show that agents often produce methodological hallucinations: silently reducing datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method's claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.26753 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.662Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).