Metraipolis › archive › dc_b88ea94dd4ff
A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications — arXiv:2608.16893
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.16893
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T14:28:58.855Z
- SHA-256 of the archived text
2fefaae4b5d28d1f184a280b69f4b88a40978561253b9eb2fa771a7728470a7c- Stable URL of this page
https://metraipolis.com/documento/dc_b88ea94dd4ff
How to cite
A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications — arXiv:2608.16893 — arXiv (en), https://arxiv.org/abs/2608.16893 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b88ea94dd4ff (Metraipolis archive dc_b88ea94dd4ff, sha256 2fefaae4b5d28d1f184a280b69f4b88a40978561253b9eb2fa771a7728470a7c)
BibTeX
@misc{dcb88ea94dd4ff,
title = {A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications — arXiv:2608.16893},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.16893},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b88ea94dd4ff; sha256 2fefaae4b5d28d1f184a280b69f4b88a40978561253b9eb2fa771a7728470a7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16893. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Expert surveys are widely used in security research to study practitioner workows and decision-making, yet recruiting domain experts - especially in Security Operations Centres (SOCs), where analysts face high workload, burnout and confidentiality constraints - is difficult and often results in small samples. Large language models (LLMs) oer an appealing alternative by generating synthetic responses at scale, but little guidance exists on when such surrogate participants are reliable. We present a methodological framework for evaluating LLMs as substitutes or supplements to expert survey respondents. Using responses from SOC professionals, we compare persona-based and aggregate LLM-generated answers across multiple models and prompting settings. We measure stability, inter-model agreement and alignment with human responses. Our results show that although LLMs produce internally consistent answers, they systematically diverge from experts, exhibiting reduced variance, central tendency bias and homogenised opinions. This work contributes methodological evidence and practical guidance to the security research community on the appropriate use and limitations of LLM-generated survey responses. We conclude that LLMs are useful for piloting and hypothesis generation but not for replacing expert elicitation, and we discuss implications for researchers using LLM-augmented surveys. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.AI). Categorie arXiv: Computers and Society (cs.CY) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.16893 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T14:27:54.335Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).