Metraipolis › archive › dc_174c871ea888
Understanding and Mitigating Role Spoofing in LLMs — arXiv:2610.40396
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40396
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:14:54.108Z
- SHA-256 of the archived text
ab093537ca9b87f5a4a3e74c5b320603d235bc359ba4bc6945303de57228a958- Stable URL of this page
https://metraipolis.com/documento/dc_174c871ea888
How to cite
Understanding and Mitigating Role Spoofing in LLMs — arXiv:2610.40396 — arXiv (en), https://arxiv.org/abs/2610.40396 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_174c871ea888 (Metraipolis archive dc_174c871ea888, sha256 ab093537ca9b87f5a4a3e74c5b320603d235bc359ba4bc6945303de57228a958)
BibTeX
@misc{dc174c871ea888,
title = {Understanding and Mitigating Role Spoofing in LLMs — arXiv:2610.40396},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40396},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_174c871ea888; sha256 ab093537ca9b87f5a4a3e74c5b320603d235bc359ba4bc6945303de57228a958},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40396. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.40396 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:08.770Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).