Metraipolis › archive › dc_49fa864f9715

Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment — arXiv:2611.11796

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.11796
License (as archived)
CC0-1.0
Archived
2026-11-10T18:43:40.399Z
SHA-256 of the archived text
408b4bdffd544501e3e08c9a9c559ff7e74d404a05808c4fa1378f8ec4c8de4e
Stable URL of this page
https://metraipolis.com/documento/dc_49fa864f9715

How to cite

Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment — arXiv:2611.11796 — arXiv (en), https://arxiv.org/abs/2611.11796 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_49fa864f9715 (Metraipolis archive dc_49fa864f9715, sha256 408b4bdffd544501e3e08c9a9c559ff7e74d404a05808c4fa1378f8ec4c8de4e)
BibTeX
@misc{dc49fa864f9715,
  title = {Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment — arXiv:2611.11796},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11796},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_49fa864f9715; sha256 408b4bdffd544501e3e08c9a9c559ff7e74d404a05808c4fa1378f8ec4c8de4e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11796. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Genomics (q-bio.GN) ; Artificial Intelligence (cs.AI); Applications (stat.AP).
Pagina del lavoro: https://arxiv.org/abs/2611.11796 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.659Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).