Metraipolis › archive › dc_ea08dfed4885

AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories — arXiv:2609.06783

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.06783
License (as archived): CC0-1.0
Archived: 2026-10-16T12:56:43.866Z
SHA-256 of the archived text: 58122cd1b160550894389f5c4a18a9bdbf7faf9d8959696c2139551e1076f007
Stable URL of this page: https://metraipolis.com/documento/dc_ea08dfed4885

How to cite

AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories — arXiv:2609.06783 — arXiv (en), https://arxiv.org/abs/2609.06783 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea08dfed4885 (Metraipolis archive dc_ea08dfed4885, sha256 58122cd1b160550894389f5c4a18a9bdbf7faf9d8959696c2139551e1076f007)

BibTeX

@misc{dcea08dfed4885,
  title = {AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories — arXiv:2609.06783},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06783},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea08dfed4885; sha256 58122cd1b160550894389f5c4a18a9bdbf7faf9d8959696c2139551e1076f007},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06783. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score, obscuring risk recognition, pre-action detection, and safe task completion when a safe solution exists. We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories. Its pipeline identifies safety-critical decision points, generates controlled variations, and constructs counterparts differing in whether a request has a safe fulfillment path. Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models. We developed rubrics to classify risk detection, action strategy, and scenario-specific action safety. Our results show that LLM agents engage in unsafe behavior more often when no safe fulfillment path exists. In these cases, frontier proprietary models more often recognize risk and exhibit safer behavior by proposing alternatives, while evaluated open-weight models more often directly execute unsafe requests. Increasing impact or reducing opportunities for oversight before execution also exposes greater vulnerability across models.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.06783 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.286Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).