Metraipolis › archive › dc_bc08b4aced65

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews — arXiv:2608.14551

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14551
License (as archived)
CC0-1.0
Archived
2026-10-18T18:56:16.761Z
SHA-256 of the archived text
a517388bb57ad32df231046516b00bc40827466e9138ffcfe48d9f8bf1440bed
Stable URL of this page
https://metraipolis.com/documento/dc_bc08b4aced65

How to cite

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews — arXiv:2608.14551 — arXiv (en), https://arxiv.org/abs/2608.14551 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc08b4aced65 (Metraipolis archive dc_bc08b4aced65, sha256 a517388bb57ad32df231046516b00bc40827466e9138ffcfe48d9f8bf1440bed)
BibTeX
@misc{dcbc08b4aced65,
  title = {Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews — arXiv:2608.14551},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14551},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc08b4aced65; sha256 a517388bb57ad32df231046516b00bc40827466e9138ffcfe48d9f8bf1440bed},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14551. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty. We show that an auxiliary BERT+GCN classifier supplies a structured uncertainty signal that improves LLM screening efficiency, and we identify the prompt-delivery strategy that maximises the benefit-to-cost ratio. We evaluate five LLM prompt-delivery conditions on eight drug-class datasets from the Cohen (2006) benchmark using 3 seeds x 5-fold stratified cross-validation (600 fold-level results). A BERT+GCN model trained per fold classifies each test paper as INCLUDE, EXCLUDE, or MAYBE via two spectral tests (algebraic radical and categorical paradox). Conditions vary information content (none / label / full scores), selectivity (all papers vs. MAYBE only), and timing (proactive vs. reactive two-pass). A cross-model pilot against gpt-4.1-mini on three datasets tests cross-generation transfer. Three findings: (i) Full-context delivery yields significant gains in F1 (+0.011, paired Wilcoxon p=0.008) and WSS@95 (+0.050, p=0.039) at a 1.28x token-cost premium, while preserving recall. (ii) MAYBE-only routing is Pareto-optimal: highest mean recall (0.92) and AUC-ROC (0.54) at only 1.05x baseline cost -- one sixth of full-context overhead. (iii) The two-pass design escalates 22.2% +/- 8.8% of records yet never revises its decision (0% flip rate across all datasets and folds), giving decisive evidence that current instruction-tuned LLMs cannot self-triage. The cross-model pilot shows an identical +0.8% recall uplift for both LLM generations. A per-paper ablation across 20,796 observations shows the dual paradox test reduces empirically to a one-line logit-gap criterion. We release the full pipeline; the 600-run experiment replays in under one hour from cached LLM responses.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Digital Libraries (cs.DL); Information Retrieval (cs.IR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.14551 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T18:06:57.825Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).