Metraipolis › archive › dc_bca7590d5b01

UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents — arXiv:2610.05622

Source
arXiv
Original
https://arxiv.org/abs/2610.05622
License (as archived)
CC0-1.0
Archived
2026-10-06T05:27:04.994Z
SHA-256 of the archived text
c9ffaecfef57a6bf4661abad5e27c2b75134aaa3aec34f2a5f7b437c9a9d2d30
Stable URL of this page
https://metraipolis.com/documento/dc_bca7590d5b01

How to cite

UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents — arXiv:2610.05622 — arXiv, https://arxiv.org/abs/2610.05622 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bca7590d5b01 (Metraipolis archive dc_bca7590d5b01, sha256 c9ffaecfef57a6bf4661abad5e27c2b75134aaa3aec34f2a5f7b437c9a9d2d30)
BibTeX
@misc{dcbca7590d5b01,
  title = {UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents — arXiv:2610.05622},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05622},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bca7590d5b01; sha256 c9ffaecfef57a6bf4661abad5e27c2b75134aaa3aec34f2a5f7b437c9a9d2d30},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05622. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence-recovery separation. Evaluations across complementary execution boundaries show that recovery is phase-dependent: before mutation, methods perform similarly without duplicate effects among capable trials; during partial mutation, naive retry, per-call idempotency, and zero-privilege journaling collapse on the evaluated composite workflows; after commit but before acknowledgment, verification and server-side idempotency substantially improve safety. These findings demonstrate that evaluating nominal completion alone masks critical, phase-dependent recovery vulnerabilities in autonomous agents.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.05622 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.057Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).