Metraipolis › archive › dc_161de28db9a4

Disentangling Task Difficulty from Run-Level Failure in Agent Failure Prediction — arXiv:2610.05572

Source
arXiv
Original
https://arxiv.org/abs/2610.05572
License (as archived)
CC0-1.0
Archived
2026-10-06T05:31:11.230Z
SHA-256 of the archived text
3184635a741ec0842686ad515ba48e6e549ab998074c0720462791e259ac72f1
Stable URL of this page
https://metraipolis.com/documento/dc_161de28db9a4

How to cite

Disentangling Task Difficulty from Run-Level Failure in Agent Failure Prediction — arXiv:2610.05572 — arXiv, https://arxiv.org/abs/2610.05572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_161de28db9a4 (Metraipolis archive dc_161de28db9a4, sha256 3184635a741ec0842686ad515ba48e6e549ab998074c0720462791e259ac72f1)
BibTeX
@misc{dc161de28db9a4,
  title = {Disentangling Task Difficulty from Run-Level Failure in Agent Failure Prediction — arXiv:2610.05572},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05572},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_161de28db9a4; sha256 3184635a741ec0842686ad515ba48e6e549ab998074c0720462791e259ac72f1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Predicting whether an LLM agent will fail has emerged as a promising direction for supporting intervention during execution. Recent approaches report strong predictive performance, often with AUROC values between 0.85 and 0.94. However, predictors are typically trained by pooling runs from many tasks. We hypothesize that part of this performance comes from recognizing that some tasks are harder than others, rather than detecting whether a particular run is heading toward failure. This distinction matters because task-level difficulty supports decisions about where to allocate computation, while run-level prediction is needed to decide whether to intervene in an ongoing trajectory. We study benchmarks with repeated attempts of the same task by the same agent and separate cross-unit comparisons from comparisons between successful and failed runs of the same model-task unit. Across the evaluated corpora, more than 99.93% of the positive-negative pairs underlying pooled AUROC are cross-unit. Accordingly, predictors that never observe the current run can achieve high pooled performance, including a difficulty oracle with AUROC up to 0.945, while remaining at chance within task. Early run-level discrimination is consistently weak across trajectory predictors, released monitors, and hidden-state probes, although it improves later in execution and is stronger for weaker agents. Under fixed token budgets, task-level allocation outperforms abort-only strategies, while early stopping becomes beneficial only when within-task AUROC reaches about 0.84-0.93, far above the 0.50-0.55 range observed for early monitors. These results show that failure prediction should be evaluated not only by outcome accuracy, but by whether the captured signal supports the intended deployment decision.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.05572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.810Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).