Metraipolis › archive › dc_8bea88560b2b
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge — arXiv:2608.21021
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.21021
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T10:18:28.416Z
- SHA-256 of the archived text
ce29e8b77556e2776da9793e43458784f6ecc6a64f4c3cca5043650a838e42ef- Stable URL of this page
https://metraipolis.com/documento/dc_8bea88560b2b
How to cite
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge — arXiv:2608.21021 — arXiv (en), https://arxiv.org/abs/2608.21021 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bea88560b2b (Metraipolis archive dc_8bea88560b2b, sha256 ce29e8b77556e2776da9793e43458784f6ecc6a64f4c3cca5043650a838e42ef)
BibTeX
@misc{dc8bea88560b2b,
title = {Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge — arXiv:2608.21021},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.21021},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bea88560b2b; sha256 ce29e8b77556e2776da9793e43458784f6ecc6a64f4c3cca5043650a838e42ef},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21021. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations and recommended actions. LLMs have emerged as a promising approach to automating this, yet whether lightweight, edge-deployable models are capable of performing in-depth free-text diagnostics remains an open question. While existing benchmarks rely on restrictive MCQs with fixed answer keys, this paper evaluates 5G domain understanding and fault analysis in a free-text generation format. Transitioning to this paradigm requires evaluating lightweight, edge-deployable AI models on open-ended diagnostic reasoning, alongside a dependable framework to validate these text outputs at scale. To address this we evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORAN FT, 5G-Faults FT, and TeleInter FT. Three independent frontier judges score outputs, and pairwise inter-judge agreement is measured as an empirical test of the LLM-as-Judge methodology. All three models reach at least 90% accuracy on fault diagnosis, while zero-shot recall of 3GPP and O-RAN specifications remains the critical gap, with all models scoring below 60%. Mean inter-judge agreement is at least 0.90 across all runs, indicating that multi-judge LLM scoring produces consistent, reproducible grades for open-ended telecom responses. Operationally, Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest inference cost and latency, making it the most suitable candidate for production telecom deployments. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Networking and Internet Architecture (cs.NI). Pagina del lavoro: https://arxiv.org/abs/2608.21021 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T10:17:37.593Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).