Metraipolis › archive › dc_4ac96cbaeca9

Evaluating LLM Generated Detection Rules in Cybersecurity — arXiv:2509.16749

Source
arXiv (en)
Original
https://arxiv.org/abs/2509.16749
License (as archived)
CC0-1.0
Archived
2026-10-19T05:15:36.901Z
SHA-256 of the archived text
04d1fad476aee049274c792f2d5d2cf8950f2b7894a43e8aca71cfbe61f68d31
Stable URL of this page
https://metraipolis.com/documento/dc_4ac96cbaeca9

How to cite

Evaluating LLM Generated Detection Rules in Cybersecurity — arXiv:2509.16749 — arXiv (en), https://arxiv.org/abs/2509.16749 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4ac96cbaeca9 (Metraipolis archive dc_4ac96cbaeca9, sha256 04d1fad476aee049274c792f2d5d2cf8950f2b7894a43e8aca71cfbe61f68d31)
BibTeX
@misc{dc4ac96cbaeca9,
  title = {Evaluating LLM Generated Detection Rules in Cybersecurity — arXiv:2509.16749},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2509.16749},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4ac96cbaeca9; sha256 04d1fad476aee049274c792f2d5d2cf8950f2b7894a43e8aca71cfbe61f68d31},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2509.16749. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLMs are increasingly pervasive in the security environment, with limited measures of their effectiveness, which limits trust and usefulness to security practitioners. Here, we present an open-source evaluation framework and benchmark metrics for evaluating LLM-generated cybersecurity rules. The benchmark employs a holdout set-based methodology to measure the effectiveness of LLM-generated security rules in comparison to a human-generated corpus of rules. It provides three key metrics inspired by the way experts evaluate security rules, offering a realistic, multifaceted evaluation of the effectiveness of an LLM-based security rule generator. This methodology is illustrated using rules from Sublime Security's detection team and those written by Sublime Security's Automated Detection Engineer (ADE), with a thorough analysis of ADE's skills presented in the results section.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2509.16749 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.660Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).