Metraipolis › archive › dc_8cd53fc422a5

Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy — arXiv:2609.35900

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35900
License (as archived)
CC0-1.0
Archived
2026-10-07T16:21:21.799Z
SHA-256 of the archived text
71e3df198544c442c72c7d4aa20bccab27610567785068b8ac8ac4e89b62ea77
Stable URL of this page
https://metraipolis.com/documento/dc_8cd53fc422a5

How to cite

Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy — arXiv:2609.35900 — arXiv (en), https://arxiv.org/abs/2609.35900 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8cd53fc422a5 (Metraipolis archive dc_8cd53fc422a5, sha256 71e3df198544c442c72c7d4aa20bccab27610567785068b8ac8ac4e89b62ea77)
BibTeX
@misc{dc8cd53fc422a5,
  title = {Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy — arXiv:2609.35900},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35900},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8cd53fc422a5; sha256 71e3df198544c442c72c7d4aa20bccab27610567785068b8ac8ac4e89b62ea77},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35900. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The integration of large language models (LLMs) into scientific workflows is accelerating, yet their ability to reconstruct the reasoning underlying published research remains unexplored. Papers specify explicit procedures while leaving many methodological dependencies-data selection, calibration corrections, priors, and domain assumptions-implicit. This ambiguity complicates the evaluation of LLM-based agents, since a failure to reproduce a result may reflect either limitations of the agent or underspecification in the source. We present a framework that evaluates agents through end-to-end reproduction, separating execution from verification and computational failure from methodological ambiguity. We apply it to fourteen astronomy studies: a case study from The Astrophysical Journal and thirteen papers published in Nature. Eleven of the thirteen contained an ambiguity preventing a uniquely specified reproduction path. In a controlled case study, twelve predefined paths, a 3x2x2 sensitivity analysis over sample definition, sky masking, and parallax zero-point treatment-gave estimates from 2.16 to 3.53 kpc for the same quantity, with only one recovering the published value (about 2.70 kpc). The published value was never used as an optimization target, selection criterion, or stopping condition; the matching path was found only after all twelve had run. Crucially, the decisive information (a +0.02 mas parallax zero-point correction) was already in the paper, but the agents did not recognize its causal relevance until the analysis made the effect visible. Matching a published outcome therefore does not validate reconstruction of the underlying reasoning, and the bottleneck is as often a failure to connect relevant information as to retrieve it. End-to-end reproduction thus serves both as a test of reproducibility and as a framework for evaluating implicit scientific knowledge in AI systems.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.MA).
Categorie arXiv: Instrumentation and Methods for Astrophysics (astro-ph.IM) ; Astrophysics of Galaxies (astro-ph.GA); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.35900 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:11.816Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).