Metraipolis › archive › dc_561bb23d0c6a

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit — arXiv:2608.16349

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.16349
License (as archived)
CC0-1.0
Archived
2026-10-18T17:43:13.955Z
SHA-256 of the archived text
0a2bb9a8ffad5e5b50f2e7a61a9e402b861baa3768efead18c39d4c6dd8c5580
Stable URL of this page
https://metraipolis.com/documento/dc_561bb23d0c6a

How to cite

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit — arXiv:2608.16349 — arXiv (en), https://arxiv.org/abs/2608.16349 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_561bb23d0c6a (Metraipolis archive dc_561bb23d0c6a, sha256 0a2bb9a8ffad5e5b50f2e7a61a9e402b861baa3768efead18c39d4c6dd8c5580)
BibTeX
@misc{dc561bb23d0c6a,
  title = {AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit — arXiv:2608.16349},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.16349},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_561bb23d0c6a; sha256 0a2bb9a8ffad5e5b50f2e7a61a9e402b861baa3768efead18c39d4c6dd8c5580},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16349. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6\%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.16349 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:13.862Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).