Metraipolis › archive › dc_b6edc5ee2a74

The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02689
License (as archived)
CC0-1.0
Archived
2026-11-03T11:08:20.271Z
SHA-256 of the archived text
d4197187aeac068170f04d1eccac6a3615f2a19a589472cebf0a4635841ccfa4
Stable URL of this page
https://metraipolis.com/documento/dc_b6edc5ee2a74

How to cite

The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689 — arXiv (en), https://arxiv.org/abs/2611.02689 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b6edc5ee2a74 (Metraipolis archive dc_b6edc5ee2a74, sha256 d4197187aeac068170f04d1eccac6a3615f2a19a589472cebf0a4635841ccfa4)
BibTeX
@misc{dcb6edc5ee2a74,
  title = {The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants — arXiv:2611.02689},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02689},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b6edc5ee2a74; sha256 d4197187aeac068170f04d1eccac6a3615f2a19a589472cebf0a4635841ccfa4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02689. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Companies deploy large language model (LLM) assistants on their websites to advise customers. For commercial reasons, the operator can give an assistant a goal that the customer never sees. This paper focuses on operator-induced vulnerability-targeted manipulation (VTM), in which an assistant pursues such a goal by exploiting a vulnerability the customer discloses. While existing research mainly measures how manipulative a model is, we go further and ask whether VTM can be detected from the customer-AI conversation. To investigate this at scale, we introduce GAP (Goal-Anchored Probing). GAP first records the option a model recommends without any goal and then sets the other option as the operator's goal. We apply GAP to 15 open- and closed-source LLMs and collect 619,600 labeled replies across 540 everyday scenarios in 12 domains. We organize our study around three research questions. (1) Do assistants with an operator goal target vulnerable customers? (2) Can VTM be detected from the customer-AI conversation? (3) Does detection generalize to unseen models, vulnerabilities, and instruction styles? We first find that 14 of 15 models push the operator goal, and 12 of them use disclosed distress or financial strain to support it. Next, we show that fine-tuned detectors identify exploitative pushes with 79\% accuracy from the reply alone, while zero-shot LLM judges average at most 62\%. Finally, detection weakens on vulnerabilities absent from training. Telling whether a push comes from the operator also becomes unreliable when the assistant is told to hide its goal. We will release our dataset and code to facilitate future research on auditing LLM assistants.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02689 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.120Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).