Metraipolis › archive › dc_3a4bd2d612b3

CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents — arXiv:2610.08871

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08871
License (as archived)
CC0-1.0
Archived
2026-10-08T11:10:14.362Z
SHA-256 of the archived text
e9718a0d9d20970865cf17fee2688ce34377a9b2b6932f5305d501fced95292e
Stable URL of this page
https://metraipolis.com/documento/dc_3a4bd2d612b3

How to cite

CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents — arXiv:2610.08871 — arXiv (en), https://arxiv.org/abs/2610.08871 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3a4bd2d612b3 (Metraipolis archive dc_3a4bd2d612b3, sha256 e9718a0d9d20970865cf17fee2688ce34377a9b2b6932f5305d501fced95292e)
BibTeX
@misc{dc3a4bd2d612b3,
  title = {CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents — arXiv:2610.08871},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08871},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3a4bd2d612b3; sha256 e9718a0d9d20970865cf17fee2688ce34377a9b2b6932f5305d501fced95292e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08871. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language model agents are increasingly deployed to automate everyday digital chores from managing emails and social media to handling banking and bills allowing users to step away from supervision. However, this capability also exposes sensitive information to phishing. Safe execution requires distinguishing malicious requests from genuine ones without simply refusing to act. Despite its practical importance, this problem remains underexplored and it is unclear whether current agents or existing defenses can achieve it. To study this problem, we first propose CredLeak-Bench, a comprehensive benchmark designed to evaluate how effectively and securely agents automate human workflows when confronted with phishing and identity verification. The benchmark covers both user-directed authentication and autonomous inbox monitoring, where agents are not explicitly instructed to log in. It systematically varies deceptive cues and pairs phishing scenarios with legitimate counterparts, enabling joint evaluation of information leakage and utility on genuine tasks. Within a sandboxed environment, leakage is measured through actual submissions of information rather than agents' self-reported behavior. Our evaluation reveals that all tested models are vulnerable to leakage. Agents also disclose sensitive information during autonomous inbox monitoring, demonstrating that phishing can induce disclosure without a user request to authenticate. Furthermore, most evaluated mitigations that reduce leakage also impair performance on genuine tasks, exposing a security utility trade off in existing defenses. These findings show why reducing leakage alone is insufficient: effective defenses must prevent unauthorized disclosure while preserving legitimate task completion. CredLeak-Bench provides a controlled framework for measuring both objectives and evaluating progress toward secure, useful agents.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.08871 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.018Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).