Metraipolis › archive › dc_4dbbe541b5a0

Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report — arXiv:2609.21843

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21843
License (as archived)
CC0-1.0
Archived
2026-10-10T02:04:11.246Z
SHA-256 of the archived text
7792ccfbfbbf89960b6486b03378cd26d8b838d0f0a8c4a10c33f4d2af50d14f
Stable URL of this page
https://metraipolis.com/documento/dc_4dbbe541b5a0

How to cite

Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report — arXiv:2609.21843 — arXiv (en), https://arxiv.org/abs/2609.21843 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4dbbe541b5a0 (Metraipolis archive dc_4dbbe541b5a0, sha256 7792ccfbfbbf89960b6486b03378cd26d8b838d0f0a8c4a10c33f4d2af50d14f)
BibTeX
@misc{dc4dbbe541b5a0,
  title = {Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report — arXiv:2609.21843},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21843},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4dbbe541b5a0; sha256 7792ccfbfbbf89960b6486b03378cd26d8b838d0f0a8c4a10c33f4d2af50d14f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21843. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This study examines tool-augmented Large Language Model (LLM) systems for supporting Root Cause Analysis (RCA) of nightly test failures at Westermo Network Technologies AB. Nightly test executions produce heterogeneous test data and logs that practitioners currently inspect manually across multiple sources. We implemented an RCA workflow in single-agent and orchestrated multi-agent configurations, both with access to test metadata and logs. An exploratory industrial case study used two real failure scenarios. Six practitioners evaluated the scenario reports through a survey and focus group, and operational measurements were collected from 120 repeated executions. The evaluation covered practitioner-perceived correctness, reasoning quality, fix realism, clarity, usefulness, and trust, as well as cost, duration, and consistency. Neither configuration showed a consistent practitioner-perceived quality advantage across the two scenarios. The single agent system generated reports faster and at lower cost, making it the more practical baseline in this context. The potential benefits of agent architectures require further evaluation in more complex scenarios.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-21 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.21843 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:43:39.837Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).