Metraipolis › archive › dc_522b360bb3c7
Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40444
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:10:48.709Z
- SHA-256 of the archived text
c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8- Stable URL of this page
https://metraipolis.com/documento/dc_522b360bb3c7
How to cite
Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444 — arXiv (en), https://arxiv.org/abs/2610.40444 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_522b360bb3c7 (Metraipolis archive dc_522b360bb3c7, sha256 c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8)
BibTeX
@misc{dc522b360bb3c7,
title = {Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40444},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_522b360bb3c7; sha256 c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40444. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Tool-use benchmarks often score agents by final task success, but terminal outcomes can obscure whether the intermediate tool-call trajectory was executable. We propose prefix executability as a trajectory-level evaluation protocol for tool-using agents. The protocol measures whether every prefix of a predicted tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies, and summarizes reliability using Prefix Executability Curves (PEC), First-Error Depth (FED), Trajectory Executability Rate (TER), horizon-normalized AUC-PEC, and Recovery Rate (RR). We instantiate the protocol on stratified samples from BFCL-v3 multi-turn and ToolSandbox, and on the complete Airline/Retail base splits of $\tau^3$-bench. We observe that outcome-only and prefix-level rankings diverge: agents whose execution is conditioned on an unverified plan often achieve higher point-estimate final success than non-plan-conditioned baselines while substantially lowering TER and AUC-PEC. These divergences are precisely the failure modes that prefix executability is designed to expose: they reveal whether apparent success rests on an executable trajectory, where invalidity first enters, and whether success follows recovery from an earlier failure. As a validation intervention, explicit pre-execution plan checking can improve executable-prefix survival while maintaining competitive terminal success. These results show that final success alone is insufficient for evaluating tool-use agents and that prefix-level executability provides an actionable reliability signal for comparing agents, diagnosing failures, and assessing validation mechanisms. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.40444 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:08.562Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).