Metraipolis › archive › dc_d496aa26e37d

Mining Meaning: Measurement Error in AI-Assisted Literature Reviews — arXiv:2609.27686

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27686
License (as archived)
CC0-1.0
Archived
2026-10-08T10:08:41.548Z
SHA-256 of the archived text
04e3d0113139e7c94bcf97975503bb988f5264b4d6a047e153f7b7d25e30b953
Stable URL of this page
https://metraipolis.com/documento/dc_d496aa26e37d

How to cite

Mining Meaning: Measurement Error in AI-Assisted Literature Reviews — arXiv:2609.27686 — arXiv (en), https://arxiv.org/abs/2609.27686 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d496aa26e37d (Metraipolis archive dc_d496aa26e37d, sha256 04e3d0113139e7c94bcf97975503bb988f5264b4d6a047e153f7b7d25e30b953)
BibTeX
@misc{dcd496aa26e37d,
  title = {Mining Meaning: Measurement Error in AI-Assisted Literature Reviews — arXiv:2609.27686},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27686},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d496aa26e37d; sha256 04e3d0113139e7c94bcf97975503bb988f5264b4d6a047e153f7b7d25e30b953},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27686. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Researchers increasingly use generative AI, particularly large language models (LLMs), to automate tasks across the research pipeline. We study the reliability of these tools at the reading, classification, and synthesis of large bodies of academic literature. We frame LLM-assisted literature reviews as a measurement problem, treating models as measurement systems and tracing how their errors affect downstream conclusions. As a test case, we use three different implementations of ChatGPT to identify and extract metadata from economics papers that use rainfall as an instrumental variable. We benchmark each implementation against a subset of human-labeled evaluation data, and then deploy those implementations to extract metadata from the full corpus. The LLMs perform well on binary classification, but performance deteriorates as tasks demand greater contextual interpretation. More importantly, how much researchers can rely on model outputs depends not only on the complexity of the reading task but also on the type of claims the data is asked to support. The same amount of measurement error substantially affects paper-level claims while having little effect on broader claims about the literature. Measurement error in LLM-generated data is thus most consequential at precisely the level of detail that constitutes an LLM's principal value added over human reviewers. We conclude that standard model performance metrics are informative about the quality of generated data but do not by themselves establish the credibility of downstream inference. Researchers must also evaluate whether substantive claims are robust to the measurement system used to generate the underlying data.

Tema degli esploratori: economia dei servizi IA (retro-catalogo arXiv: annunci del 2026-09-24 in econ.GN).
Categorie arXiv: General Economics (econ.GN).
Pagina del lavoro: https://arxiv.org/abs/2609.27686 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:17:59.700Z dalla pagina https://arxiv.org/catchup/econ.GN/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).