Metraipolis › archive › dc_4280e76eccb5

SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems — arXiv:2609.37294

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37294
License (as archived)
CC0-1.0
Archived
2026-10-07T17:32:51.378Z
SHA-256 of the archived text
298fa8e5c506e953371771ce3ac3c47677adb55032c1aa3f94c7b9076a1f22e9
Stable URL of this page
https://metraipolis.com/documento/dc_4280e76eccb5

How to cite

SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems — arXiv:2609.37294 — arXiv (en), https://arxiv.org/abs/2609.37294 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4280e76eccb5 (Metraipolis archive dc_4280e76eccb5, sha256 298fa8e5c506e953371771ce3ac3c47677adb55032c1aa3f94c7b9076a1f22e9)
BibTeX
@misc{dc4280e76eccb5,
  title = {SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems — arXiv:2609.37294},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37294},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4280e76eccb5; sha256 298fa8e5c506e953371771ce3ac3c47677adb55032c1aa3f94c7b9076a1f22e9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37294. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibility, and comparability of their evaluations. Conventional practices such as reporting a single output, an average score, best-of-N, or pass@k performance can obscure variability and estimation uncertainty, potentially leading to misleading conclusions about system reliability. This article presents SafeLLM4SE, a practical methodology and reporting standard for statistically principled evaluation of LLM-based software engineering systems. Rather than treating generated outputs as deterministic artifacts, SafeLLM4SE treats them as realizations of a stochastic process and distinguishes quality, stability, and estimation uncertainty. It combines adaptive sampling with confidence intervals, distribution-aware statistical comparisons, effect sizes, and a minimum reporting standard covering model configuration, reproducibility, evaluation procedures, and resource usage. SafeLLM4SE is also provided as an open-source software package available on PyPI, enabling researchers and practitioners to reproduce and extend the methodology. We illustrate its application by comparing two LLMs on HumanEval, a benchmark of programming problems assessed through functional tests.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Software Engineering (cs.SE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.37294 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.535Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).