Metraipolis › archive › dc_caaf151298ce
BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories — arXiv:2611.06911
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.06911
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-05T18:17:15.292Z
- SHA-256 of the archived text
dad94129f1ea41b376b8e286d7ebad4be940b5dc7d6ae7f1478e163ebef58399- Stable URL of this page
https://metraipolis.com/documento/dc_caaf151298ce
How to cite
BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories — arXiv:2611.06911 — arXiv (en), https://arxiv.org/abs/2611.06911 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_caaf151298ce (Metraipolis archive dc_caaf151298ce, sha256 dad94129f1ea41b376b8e286d7ebad4be940b5dc7d6ae7f1478e163ebef58399)
BibTeX
@misc{dccaaf151298ce,
title = {BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories — arXiv:2611.06911},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.06911},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_caaf151298ce; sha256 dad94129f1ea41b376b8e286d7ebad4be940b5dc7d6ae7f1478e163ebef58399},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06911. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.06911 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-05T18:12:49.021Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).