Metraipolis › archive › dc_83d398d4252f

Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation — arXiv:2609.25173

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25173
License (as archived)
CC0-1.0
Archived
2026-10-08T14:41:54.232Z
SHA-256 of the archived text
92674d96682bda89f3a8e8a96adc087e66c16d858d7a7ce11ed5ebcce8415df9
Stable URL of this page
https://metraipolis.com/documento/dc_83d398d4252f

How to cite

Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation — arXiv:2609.25173 — arXiv (en), https://arxiv.org/abs/2609.25173 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_83d398d4252f (Metraipolis archive dc_83d398d4252f, sha256 92674d96682bda89f3a8e8a96adc087e66c16d858d7a7ce11ed5ebcce8415df9)
BibTeX
@misc{dc83d398d4252f,
  title = {Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation — arXiv:2609.25173},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25173},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_83d398d4252f; sha256 92674d96682bda89f3a8e8a96adc087e66c16d858d7a7ce11ed5ebcce8415df9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25173. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Attack success rate (ASR) is the headline metric in nearly every published evaluation of attacks on, and defenses for, LLM agents. We argue that ASR as currently used is not a single quantity but a family of metrics parameterized by six design choices that papers seldom specify and never hold constant across the literature. We support this with two studies that require no proprietary access. First, a full-text meta-analysis of 259 agentic-security papers posted to arXiv between February 2025 and September 2026 finds that most report neither a variance estimate nor repeated runs for their headline attack metric: 58% (95% CI 44-71) in a hand-coded random sample of 50, 65.3% by automated coding of all 259. Only 30.9% disclose enough about decoding to establish whether their evaluation was even stochastic, and of the 64 papers we confirm use an LLM judge, 29.7% report any agreement check against human labels. Second, an analytical study shows that these omissions are not cosmetic: on a 100-instance benchmark, the minimum difference in ASR detectable at conventional power is 18.2 percentage points, and two defenses whose true ASRs differ by 5 points are ranked in the wrong order by a single-run evaluation roughly 21% of the time. Because several of the six axes shift ASR in a system-dependent way, the resulting incomparability is not a constant offset that cancels in comparison. We conclude that cross-paper ASR comparison is currently unsupported, and propose a ten-item reporting checklist targeted at each failure we measure. Our aim is not to dispute any individual result but to supply the shared measurement contract the field has so far done without.

Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CR).
Categorie arXiv: Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2609.25173 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:55:11.719Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).