Metraipolis › archive › dc_040eb40b1560

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations — arXiv:2608.12002

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12002
License (as archived)
CC0-1.0
Archived
2026-10-19T05:15:33.055Z
SHA-256 of the archived text
1b877e760b3e00b7086a5d4296a7528ebddf1908864e042bee2e14d41a301e08
Stable URL of this page
https://metraipolis.com/documento/dc_040eb40b1560

How to cite

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations — arXiv:2608.12002 — arXiv (en), https://arxiv.org/abs/2608.12002 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_040eb40b1560 (Metraipolis archive dc_040eb40b1560, sha256 1b877e760b3e00b7086a5d4296a7528ebddf1908864e042bee2e14d41a301e08)
BibTeX
@misc{dc040eb40b1560,
  title = {CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations — arXiv:2608.12002},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12002},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_040eb40b1560; sha256 1b877e760b3e00b7086a5d4296a7528ebddf1908864e042bee2e14d41a301e08},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12002. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.12002 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.651Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).