Metraipolis › archive › dc_b338f11b60dd

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI — arXiv:2610.00682

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00682
License (as archived)
CC0-1.0
Archived
2026-10-06T12:02:37.184Z
SHA-256 of the archived text
2b48f98b2985d638e6e2de9c19a6d0a84e0b999ae39baeae50d20714a696dd9f
Stable URL of this page
https://metraipolis.com/documento/dc_b338f11b60dd

How to cite

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI — arXiv:2610.00682 — arXiv (en), https://arxiv.org/abs/2610.00682 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b338f11b60dd (Metraipolis archive dc_b338f11b60dd, sha256 2b48f98b2985d638e6e2de9c19a6d0a84e0b999ae39baeae50d20714a696dd9f)
BibTeX
@misc{dcb338f11b60dd,
  title = {Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI — arXiv:2610.00682},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00682},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b338f11b60dd; sha256 2b48f98b2985d638e6e2de9c19a6d0a84e0b999ae39baeae50d20714a696dd9f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00682. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) increasingly underpin scientific AI applications that reason over structured knowledge, from biomedical question answering to materials informatics. However, their logical reasoning often falls short, producing factual inaccuracies unacceptable in these settings. Reliable evaluation remains challenging: manual dataset construction scales poorly, and LLM-based generation risks embedding the very flaws it aims to measure. High-quality benchmarks must ground both correct and incorrect labelled examples in explicit background knowledge, formally verifiable by a standard reasoner. We propose a pipeline that automatically generates ontology-grounded multiple-choice question (MCQ) benchmarks from any sufficiently axiomatised OWL 2 ontology, with correct answers grounded in the ontology by design. Distractors are generated by perturbing the right-hand-side class expressions of class definition axioms, and their incorrectness is formally verified by an OWL reasoner via entailment checks. We evaluate the pipeline on three ontologies: Pizza (small, academic), PMDco (complex, materials science), and DOID (large, biomedical), generating 112, 2,491, and 15,216 MCQs respectively. Distractors span four semantic categories from class unsatisfiability to weakened subsumptions, enabling diagnostic evaluation of specific reasoning failures. Items meet natural language quality standards: mean LLM judge scores of 4.02, 4.36, and 3.36 out of 5 confirm fluency, and correct-answer-to-distractor similarity above 0.8 shows that wrong options cannot be dismissed on surface form alone. Six LLMs evaluated zero-shot achieve 41.1-76.8% accuracy, well above the 25% random-guessing baseline, indicating the benchmarks are challenging and discriminative. This work is a step towards more reliable benchmarks for assessing logical reasoning in scientific AI.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.00682 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.881Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).