Metraipolis › archive › dc_baee6db057c7

RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use — arXiv:2610.15884

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.15884
License (as archived)
CC0-1.0
Archived
2026-11-06T14:51:34.653Z
SHA-256 of the archived text
b8c7ec54207fbe6af8d2968dd7db12e99d699dec8ccff5be3381fbf5b9456332
Stable URL of this page
https://metraipolis.com/documento/dc_baee6db057c7

How to cite

RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use — arXiv:2610.15884 — arXiv (en), https://arxiv.org/abs/2610.15884 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_baee6db057c7 (Metraipolis archive dc_baee6db057c7, sha256 b8c7ec54207fbe6af8d2968dd7db12e99d699dec8ccff5be3381fbf5b9456332)
BibTeX
@misc{dcbaee6db057c7,
  title = {RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use — arXiv:2610.15884},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.15884},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_baee6db057c7; sha256 b8c7ec54207fbe6af8d2968dd7db12e99d699dec8ccff5be3381fbf5b9456332},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.15884. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents that call external tools must decide what to do when a tool call times out: retrying risks duplicate side effects, while not retrying can leave the task incomplete. We present RetryBench, an evaluation protocol for retry decisions and failure recovery in tool-using LLM agents. Each decision is scored on retry permission (is the retry justified by an idempotency-key guarantee or proof of non-execution?), settled effects (duplicates and omissions), and completion, by replaying it across all hidden execution worlds consistent with the observation. On four synthetic software-engineering tasks, we run two controlled prompt-sensitivity studies (1,728 decisions). Study 1 (idempotency-key descriptions): with the contract fixed, GPT-5.6 Luna reuses the valid idempotency key in 10/40 decisions when an extra key is described as a "Candidate recovery key" but in 39/40 when it is an "Alternative key string"; across three further phrasings per family the gap is 27/120 versus 113/120. GPT-5.6 Sol and Terra are at ceiling (40/40). Study 2 (action order): under underspecified lookup guarantees, retry rates are 17-27 points lower for GPT-5.6 Terra and GPT-5.4 when retry is listed first (option-order bias), mainly on two tasks; we make no claim about which behavior is better. Findings are limited to researcher-chosen phrasings, four tasks and unpinned model aliases. Code and all model responses are released.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Cryptography and Security (cs.CR); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.15884 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:57.536Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).