Metraipolis › archive › dc_acedbbe08032

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures — arXiv:2609.27773

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.27773
License (as archived): CC0-1.0
Archived: 2026-10-08T09:12:59.538Z
SHA-256 of the archived text: 2ec21de2bbc9f5a9edb22c91e5ddeaa6166fce074f11369cfbb1397667a824c4
Stable URL of this page: https://metraipolis.com/documento/dc_acedbbe08032

How to cite

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures — arXiv:2609.27773 — arXiv (en), https://arxiv.org/abs/2609.27773 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_acedbbe08032 (Metraipolis archive dc_acedbbe08032, sha256 2ec21de2bbc9f5a9edb22c91e5ddeaa6166fce074f11369cfbb1397667a824c4)

BibTeX

@misc{dcacedbbe08032,
  title = {Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures — arXiv:2609.27773},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27773},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_acedbbe08032; sha256 2ec21de2bbc9f5a9edb22c91e5ddeaa6166fce074f11369cfbb1397667a824c4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27773. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As Large Language Models (LLMs) move from conversational assistants to advanced agentic systems, guardrail failures can convert adversarial intents into harmful executions. However, most guardrail evaluation frameworks focus only on the result and assess whether a user request is safe or unsafe. This approach is insufficient for multi-turn failures, where adversarial intent is distributed across multiple turns. This motivates us to go beyond detection to identify the turns and tokens that push the conversation toward unsafe trajectories. To support this, we construct a multi-turn dataset with behavioral validation and tiered evidence supervision. The dataset contains 1,762 conversations, including adversarial conversations, benign twins, and benign variants with high-risk vocabulary. We train a lightweight hierarchical attribution model that predicts safety violations and attributes them to contributing user turns and token spans. The model achieves strong detection performance (F1=0.988), and removing the top 15% of attributed tokens reduces the adversarial classification confidence by 51.1%. The model preserves low false positive rates on benign conversations with high-risk vocabulary, with false positives below 1% on both borderline benign and benign high-risk vocabulary conversations, compared to 37.3% and 94.7% for a keyword-based surface-risk baseline. Independent human annotation supports the model's attribution performance, with the top-five attributed turns containing a human-identified evidence-bearing turn in 84.5% of adversarial cases.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.27773 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.711Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).