Metraipolis › archive › dc_f36eac7dacc5
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry — arXiv:2608.14680
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14680
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T17:05:56.529Z
- SHA-256 of the archived text
488bc8977649471f14f6361066c62b0cac8c2b7a123bc5a8335e4de15bc9b484- Stable URL of this page
https://metraipolis.com/documento/dc_f36eac7dacc5
How to cite
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry — arXiv:2608.14680 — arXiv (en), https://arxiv.org/abs/2608.14680 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f36eac7dacc5 (Metraipolis archive dc_f36eac7dacc5, sha256 488bc8977649471f14f6361066c62b0cac8c2b7a123bc5a8335e4de15bc9b484)
BibTeX
@misc{dcf36eac7dacc5,
title = {When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry — arXiv:2608.14680},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14680},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f36eac7dacc5; sha256 488bc8977649471f14f6361066c62b0cac8c2b7a123bc5a8335e4de15bc9b484},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14680. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.14680 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:13.724Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).