Metraipolis › archive › dc_4dc36ee38c0f

Knowledge Synthesis Review Framework: Task-Level Benchmarking of LLM-Based Systems for Multi-Source Evidence Synthesis — arXiv:2608.12741

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12741
License (as archived)
CC0-1.0
Archived
2026-10-19T04:55:12.775Z
SHA-256 of the archived text
f3ba443a27d8d49800dc5b2ae37417d2b01571e57a5b70b780a7de5b0f253c73
Stable URL of this page
https://metraipolis.com/documento/dc_4dc36ee38c0f

How to cite

Knowledge Synthesis Review Framework: Task-Level Benchmarking of LLM-Based Systems for Multi-Source Evidence Synthesis — arXiv:2608.12741 — arXiv (en), https://arxiv.org/abs/2608.12741 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4dc36ee38c0f (Metraipolis archive dc_4dc36ee38c0f, sha256 f3ba443a27d8d49800dc5b2ae37417d2b01571e57a5b70b780a7de5b0f253c73)
BibTeX
@misc{dc4dc36ee38c0f,
  title = {Knowledge Synthesis Review Framework: Task-Level Benchmarking of LLM-Based Systems for Multi-Source Evidence Synthesis — arXiv:2608.12741},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12741},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4dc36ee38c0f; sha256 f3ba443a27d8d49800dc5b2ae37417d2b01571e57a5b70b780a7de5b0f253c73},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12741. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evidence in rapidly evolving fields is fragmented across academic studies, industry reports, policy documents, and media sources that differ in quality, structure, and purpose, making timely synthesis difficult. Large language models (LLMs) may accelerate this work, but their reliability across the distinct cognitive tasks of a review remains uncertain. We introduce the Knowledge Synthesis Review (KSR), a human-in-the-loop framework that decomposes evidence synthesis into screening, extraction, analysis, and synthesis, benchmarks LLM-based systems on each task against expert reference standards, and routes each task to the best-performing system under continuous expert validation. We evaluated GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, and NotebookLM on a 244-document benchmark subset drawn from a 1,893-document corpus on AI and work spanning four source types, against a gold standard with high inter-rater reliability (92.2% agreement, kappa = 0.80). No system led on all tasks. Claude Sonnet 4 achieved the highest screening accuracy (82.8%) and GPT-5 the highest recall (91.8%) at the expense of lower specificity. Extraction exceeded 90% agreement for titles and sources but degraded in author and reference fields. Performance declined most in interpretive analysis and cross-source synthesis, where expert judgment remained essential. A contamination check on post-cutoff documents showed no evidence that prior exposure inflated results. Applied to the full corpus, the routed workflow surfaced cross-source asymmetries and blind spots that single-source synthesis would miss, including worker well-being, small firms, and the Global South. KSR offers a transparent, auditable, model-agnostic framework for governing LLM assistance in research synthesis while preserving human accountability.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-14 in cs.IR).
Categorie arXiv: Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2608.12741 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:46:09.752Z dalla pagina https://arxiv.org/catchup/cs.IR/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).