Metraipolis › archive › dc_d646947e2193

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics — arXiv:2610.21074

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21074
License (as archived)
CC0-1.0
Archived
2026-10-16T08:59:41.959Z
SHA-256 of the archived text
37d82451f7cb69a7b06dcdcd7ba66b324458ab74f561144ed1229d1d4d164fab
Stable URL of this page
https://metraipolis.com/documento/dc_d646947e2193

How to cite

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics — arXiv:2610.21074 — arXiv (en), https://arxiv.org/abs/2610.21074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d646947e2193 (Metraipolis archive dc_d646947e2193, sha256 37d82451f7cb69a7b06dcdcd7ba66b324458ab74f561144ed1229d1d4d164fab)
BibTeX
@misc{dcd646947e2193,
  title = {OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics — arXiv:2610.21074},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21074},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d646947e2193; sha256 37d82451f7cb69a7b06dcdcd7ba66b324458ab74f561144ed1229d1d4d164fab},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Genomics (q-bio.GN); Quantitative Methods (q-bio.QM).
Pagina del lavoro: https://arxiv.org/abs/2610.21074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.692Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).