Metraipolis › archive › dc_48aa16ae2d8a

Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment — arXiv:2608.21057

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.21057
License (as archived)
CC0-1.0
Archived
2026-10-18T10:31:10.629Z
SHA-256 of the archived text
471acfc6addd7dad670282323785d1e87a5ca0a8e31413f9ed17a8353d748c9b
Stable URL of this page
https://metraipolis.com/documento/dc_48aa16ae2d8a

How to cite

Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment — arXiv:2608.21057 — arXiv (en), https://arxiv.org/abs/2608.21057 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48aa16ae2d8a (Metraipolis archive dc_48aa16ae2d8a, sha256 471acfc6addd7dad670282323785d1e87a5ca0a8e31413f9ed17a8353d748c9b)
BibTeX
@misc{dc48aa16ae2d8a,
  title = {Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment — arXiv:2608.21057},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.21057},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48aa16ae2d8a; sha256 471acfc6addd7dad670282323785d1e87a5ca0a8e31413f9ed17a8353d748c9b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21057. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic large language model (LLM) systems are reshaping scientific workflows in chemistry and drug discovery, but evaluating their open-ended, tool-augmented outputs remains a fundamental bottleneck. The LLM-as-a-Judge paradigm has emerged as a scalable alternative, but existing drug discovery benchmarks deploy LLM judges without validating their alignment with human experts. In this work, we present an LLM-as-a-Judge evaluation framework for ChatInvent, an agentic drug discovery assistant deployed at AstraZeneca, with five contributions. First, we define four output-quality evaluation dimensions---Completeness, Relevancy, Structural Clarity, and Scope Adherence---alongside deterministic Tool Call Correctness checks. Second, we validate the judge through a human alignment study with five expert annotators, comparing Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, and Llama 3.1 70B as candidate judges. Third, we optimize the best-performing judge using few-shot demonstrations of human-annotated examples, improving alignment with the human majority vote from 0.80 to 0.86. Fourth, applying the optimized judge to 70 held-out questions, we surface concrete limitations and find no strong evidence that informal phrasing degrades output quality; it may, however, still be helpful to have the LLM rewrite the original question before querying the agent. Finally, we extend the framework to 38 adversarial questions that are ambiguous, invalid, out-of-scope or ethically sensitive, and show that the agent's refusal behavior is guided by the stated intent of a request. Our framework provides a reusable template for human-aligned evaluation of agentic systems in scientific domains.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.21057 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T10:27:55.950Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).