Metraipolis › archive › dc_e3824510cbff

The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02689
License (as archived): CC0-1.0
Archived: 2026-11-03T10:37:51.139Z
SHA-256 of the archived text: 4bb92415a8e71440ada07c4b27ed22f077666133f82f60c52d96e65381a4be41
Stable URL of this page: https://metraipolis.com/documento/dc_e3824510cbff

How to cite

The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689 — arXiv (en), https://arxiv.org/abs/2611.02689 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e3824510cbff (Metraipolis archive dc_e3824510cbff, sha256 4bb92415a8e71440ada07c4b27ed22f077666133f82f60c52d96e65381a4be41)

BibTeX

@misc{dce3824510cbff,
  title = {The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02689},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e3824510cbff; sha256 4bb92415a8e71440ada07c4b27ed22f077666133f82f60c52d96e65381a4be41},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02689. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Companies deploy large language model (LLM) assistants on their websites to advise customers. For commercial reasons, the operator can give an assistant a goal that the customer never sees. This paper focuses on operator-induced vulnerability-targeted manipulation (VTM), in which an assistant pursues such a goal by exploiting a vulnerability the customer discloses. While existing research mainly measures how manipulative a model is, we go further and ask whether VTM can be detected from the customer-AI conversation. To investigate this at scale, we introduce GAP (Goal-Anchored Probing). GAP first records the option a model recommends without any goal and then sets the other option as the operator's goal. We apply GAP to 15 open- and closed-source LLMs and collect 619,600 labeled replies across 540 everyday scenarios in 12 domains. We organize our study around three research questions. (1) Do assistants with an operator goal target vulnerable customers? (2) Can VTM be detected from the customer-AI conversation? (3) Does detection generalize to unseen models, vulnerabilities, and instruction styles? We first find that 14 of 15 models push the operator goal, and 12 of them use disclosed distress or financial strain to support it. Next, we show that fine-tuned detectors identify exploitative pushes with 79\% accuracy from the reply alone, while zero-shot LLM judges average at most 62\%. Finally, detection weakens on vulnerabilities absent from training. Telling whether a push comes from the operator also becomes unreliable when the assistant is told to hide its goal. We will release our dataset and code to facilitate future research on auditing LLM assistants.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02689 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:29:48.888Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).