Metraipolis › archive › dc_cc63001ed64c

Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency — arXiv:2608.26885

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26885
License (as archived)
CC0-1.0
Archived
2026-10-17T14:23:09.444Z
SHA-256 of the archived text
e31bf15f159e6e73ab13a097f6e9a741f8a28ac7353d869b299af460a6095b2e
Stable URL of this page
https://metraipolis.com/documento/dc_cc63001ed64c

How to cite

Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency — arXiv:2608.26885 — arXiv (en), https://arxiv.org/abs/2608.26885 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cc63001ed64c (Metraipolis archive dc_cc63001ed64c, sha256 e31bf15f159e6e73ab13a097f6e9a741f8a28ac7353d869b299af460a6095b2e)
BibTeX
@misc{dccc63001ed64c,
  title = {Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency — arXiv:2608.26885},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26885},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cc63001ed64c; sha256 e31bf15f159e6e73ab13a097f6e9a741f8a28ac7353d869b299af460a6095b2e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26885. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Background. Large language models (LLMs) are increasingly used for screening in evidence synthesis, where false negatives can remove relevant studies before full-text assessment. We compared human and LLM title-and-abstract screening workflows in a preregistered study embedded in a conceptually complex scoping review. Methods. After a conservative title-only screen, 1,131 records were screened by one review lead, four trained assistants screening non-overlapping subsets, and seven complete LLM runs using different models and processing configurations, including a nominally identical repeat run. We compared retained workload, operational recall against 316 verified eligible records, agreement, run-to-run consistency, and procedural burden. Because eligibility was verified only for records advanced and assessed in the parent review, recall estimates were operational. Results. No workflow recovered all verified eligible records. The human workflows and two GPT-5.4 file-batch runs retained 42.2-45.0% of records while achieving 82.3-82.9% recall. Gemini 3.1 file batches achieved the highest recall (83.9%) but retained 56.7% of records. All-at-once configurations recovered fewer eligible records than corresponding file-batch configurations. Two nominally identical GPT-5.4 file-batch runs agreed on 91.7% of records but differed on 94 records, including 29 verified eligible records retained by only one run. Discussion. LLM screening performance depended on the implemented workflow, not model identity alone. Processing configuration, workload, record-level variation, and human-LLM decision integration are therefore substantive properties of deployed systems. For high-recall tasks, LLMs are better suited to validated, auditable, human-supervised workflows than autonomous exclusion.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Human-Computer Interaction (cs.HC); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.26885 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.528Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).