Metraipolis › archive › dc_522b360bb3c7

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40444
License (as archived)
CC0-1.0
Archived
2026-10-31T06:10:48.709Z
SHA-256 of the archived text
c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8
Stable URL of this page
https://metraipolis.com/documento/dc_522b360bb3c7

How to cite

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444 — arXiv (en), https://arxiv.org/abs/2610.40444 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_522b360bb3c7 (Metraipolis archive dc_522b360bb3c7, sha256 c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8)
BibTeX
@misc{dc522b360bb3c7,
  title = {Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success — arXiv:2610.40444},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40444},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_522b360bb3c7; sha256 c5a10da059fa0143127e501cc4d27c8479349efb451a2f4e3ce7a243f57f63b8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40444. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Tool-use benchmarks often score agents by final task success, but terminal outcomes can obscure whether the intermediate tool-call trajectory was executable. We propose prefix executability as a trajectory-level evaluation protocol for tool-using agents. The protocol measures whether every prefix of a predicted tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies, and summarizes reliability using Prefix Executability Curves (PEC), First-Error Depth (FED), Trajectory Executability Rate (TER), horizon-normalized AUC-PEC, and Recovery Rate (RR). We instantiate the protocol on stratified samples from BFCL-v3 multi-turn and ToolSandbox, and on the complete Airline/Retail base splits of $\tau^3$-bench. We observe that outcome-only and prefix-level rankings diverge: agents whose execution is conditioned on an unverified plan often achieve higher point-estimate final success than non-plan-conditioned baselines while substantially lowering TER and AUC-PEC. These divergences are precisely the failure modes that prefix executability is designed to expose: they reveal whether apparent success rests on an executable trajectory, where invalidity first enters, and whether success follows recovery from an earlier failure. As a validation intervention, explicit pre-execution plan checking can improve executable-prefix survival while maintaining competitive terminal success. These results show that final success alone is insufficient for evaluating tool-use agents and that prefix-level executability provides an actionable reliability signal for comparing agents, diagnosing failures, and assessing validation mechanisms.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.40444 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.562Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).