Metraipolis › archive › dc_af2c69bd6aa8

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners — arXiv:2608.27424

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27424
License (as archived)
CC0-1.0
Archived
2026-10-17T14:39:22.082Z
SHA-256 of the archived text
3a70c8ef25c188c3801c6fe087b98c5c51a95b48e92d553ce3fde7770dcf633f
Stable URL of this page
https://metraipolis.com/documento/dc_af2c69bd6aa8

How to cite

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners — arXiv:2608.27424 — arXiv (en), https://arxiv.org/abs/2608.27424 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_af2c69bd6aa8 (Metraipolis archive dc_af2c69bd6aa8, sha256 3a70c8ef25c188c3801c6fe087b98c5c51a95b48e92d553ce3fde7770dcf633f)
BibTeX
@misc{dcaf2c69bd6aa8,
  title = {Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners — arXiv:2608.27424},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27424},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_af2c69bd6aa8; sha256 3a70c8ef25c188c3801c6fe087b98c5c51a95b48e92d553ce3fde7770dcf633f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27424. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditional on making a definitive judgment, ModelScan achieved 100% precision, recall, and F1. Fickling identified no unique true- positive families beyond those found by the combination of ModelAudit and ModelScan. Furthermore, for the 48 malicious families where ModelScan failed to complete its analysis, both ModelAudit and Fickling generated detections consistent with ground truth. These findings underscore the need to separate judgment accuracy from judgment availability, as well as incremental detection coverage from tool-level redundancy.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27424 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.724Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).