Metraipolis › archive › dc_4d75ebb999bb
Are LLM-Agent Prompt-Injection Benchmarks Still Valid? A Cross-Model and Cross-Benchmark Study (2026) · DOI 10.21203/rs.3.rs-10962146/v1
- Source
- Metraipolis archive
- Original
- https://doi.org/10.21203/rs.3.rs-10962146/v1
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-08T05:45:50.276Z
- SHA-256 of the archived text
341d42763e1802fee2807ccdf0339e650f0151213d98ae7c6a69c412f92d84b3- Stable URL of this page
https://metraipolis.com/documento/dc_4d75ebb999bb
How to cite
Are LLM-Agent Prompt-Injection Benchmarks Still Valid? A Cross-Model and Cross-Benchmark Study (2026) · DOI 10.21203/rs.3.rs-10962146/v1 — Metraipolis archive, https://doi.org/10.21203/rs.3.rs-10962146/v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_4d75ebb999bb (Metraipolis archive dc_4d75ebb999bb, sha256 341d42763e1802fee2807ccdf0339e650f0151213d98ae7c6a69c412f92d84b3)
BibTeX
@misc{dc4d75ebb999bb,
title = {Are LLM-Agent Prompt-Injection Benchmarks Still Valid? A Cross-Model and Cross-Benchmark Study (2026) · DOI 10.21203/rs.3.rs-10962146/v1},
howpublished = {Metraipolis archive},
url = {https://doi.org/10.21203/rs.3.rs-10962146/v1},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_4d75ebb999bb; sha256 341d42763e1802fee2807ccdf0339e650f0151213d98ae7c6a69c412f92d84b3},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.21203/rs.3.rs-10962146/v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content). Titolo: Are LLM-Agent Prompt-Injection Benchmarks Still Valid? A Cross-Model and Cross-Benchmark Study Autori: Sushil Khairnar Pubblicato: 2026-10-07 · editore: Springer Science and Business Media LLC DOI: https://doi.org/10.21203/rs.3.rs-10962146/v1 Licenza dell'articolo: CC-BY-4.0 Abstract (testo degli autori, CC-BY-4.0): Abstract Indirect prompt injection is the leading security problem for tool-using LLM agents, and a growing number of benchmarks have been developed to measure the robustness of the agents and the effectiveness of defenses against it. This work highlights an essential question which is often ignored in the research community: Is it true that these benchmarks are still applicable for current models, and is the robustness demonstrated on one benchmark portable to another benchmark, another vendor, or to another defense strategy? Using a unified, faithful, and configurable harness, we evaluate the attack success rate (ASR) of 11 models from five different vendors (Anthropic, Meta, Amazon, Mistral, DeepSeek) on two single-shot benchmarks with different threat models - InjecAgent (tool hijacking) and a targeted BIPIA-email injection (benign content insertion) and on the agent benchmark AgentDojo for four vendors, together with the evaluation of defense on ten models. Our findings show three primary results. First, benchmark saturation is vendor-specific: InjecAgent achieves 0% ASR on the Claude family of models, while other vendors are still vulnerable to attacks at rates ranging from 16% to 76%, indicating that the benchmark is only solved for Anthropic models. Second, the two single-shot benchmarks give conflicting evaluations of the robustness of the same models (e.g., Claude Opus 5 scores 0% ASR on InjecAgent, yet 59% on BIPIA). They have negatively correlated robustness ranking (Kendall’s τ = −0.24, permutation p = 0.41, underpowered with 11 models) and five out of eleven models are rated secure on one benchmark but vulnerable on the other. Third, none of the examined prompt-based defenses works universally across models. Fonte: https://api.crossref.org/works/10.21203%2Frs.3.rs-10962146%2Fv1 — metadati Crossref, consultati il 2026-10-08. Interrogazione: «prompt injection attacks».