Metraipolis › archive › dc_eb0ee995607b

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment — arXiv:2609.11234

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.11234
License (as archived): CC0-1.0
Archived: 2026-10-16T05:23:53.772Z
SHA-256 of the archived text: abe47452b8c81a641105ac5de945d52ded49c7c6f3bd39ce3e40eb8ca3ce4d96
Stable URL of this page: https://metraipolis.com/documento/dc_eb0ee995607b

How to cite

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment — arXiv:2609.11234 — arXiv (en), https://arxiv.org/abs/2609.11234 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb0ee995607b (Metraipolis archive dc_eb0ee995607b, sha256 abe47452b8c81a641105ac5de945d52ded49c7c6f3bd39ce3e40eb8ca3ce4d96)

BibTeX

@misc{dceb0ee995607b,
  title = {NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment — arXiv:2609.11234},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.11234},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb0ee995607b; sha256 abe47452b8c81a641105ac5de945d52ded49c7c6f3bd39ce3e40eb8ca3ce4d96},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11234. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.11234 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T05:17:51.990Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).