Metraipolis › archive › dc_79f4ca2edb08
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers — arXiv:2609.11117
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.11117
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T05:40:45.462Z
- SHA-256 of the archived text
02c5060a661ea6593c00d6058cc11490907a754180514cf45d8b8cdda58810e3- Stable URL of this page
https://metraipolis.com/documento/dc_79f4ca2edb08
How to cite
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers — arXiv:2609.11117 — arXiv (en), https://arxiv.org/abs/2609.11117 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_79f4ca2edb08 (Metraipolis archive dc_79f4ca2edb08, sha256 02c5060a661ea6593c00d6058cc11490907a754180514cf45d8b8cdda58810e3)
BibTeX
@misc{dc79f4ca2edb08,
title = {Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers — arXiv:2609.11117},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.11117},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_79f4ca2edb08; sha256 02c5060a661ea6593c00d6058cc11490907a754180514cf45d8b8cdda58810e3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11117. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.11117 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T05:23:56.999Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).