Metraipolis › archive › dc_9be22f4f020b

BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents — arXiv:2609.16305

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.16305
License (as archived): CC0-1.0
Archived: 2026-10-15T18:31:18.333Z
SHA-256 of the archived text: b8ac7c9dedf302bb224ca7807b91f562d2b6d3607b1ff3a2f9c2722e867a71ff
Stable URL of this page: https://metraipolis.com/documento/dc_9be22f4f020b

How to cite

BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents — arXiv:2609.16305 — arXiv (en), https://arxiv.org/abs/2609.16305 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9be22f4f020b (Metraipolis archive dc_9be22f4f020b, sha256 b8ac7c9dedf302bb224ca7807b91f562d2b6d3607b1ff3a2f9c2722e867a71ff)

BibTeX

@misc{dc9be22f4f020b,
  title = {BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents — arXiv:2609.16305},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16305},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9be22f4f020b; sha256 b8ac7c9dedf302bb224ca7807b91f562d2b6d3607b1ff3a2f9c2722e867a71ff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16305. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) agents increasingly operate over long-horizon interactions involving tool use, persistent state, evolving authorization, and external environment feedback. In such settings, safety failures may emerge only after multiple turns, yet existing evaluations often reduce agent behavior to task or attack success, obscuring whether an agent acts, refuses, or remains appropriately calibrated as the interaction evolves. We introduce Blindspot, a benchmark for trajectory-level safety calibration of long-horizon tool-using agents. Blindspot evaluates complete user-agent-environment trajectories through adaptive adversarial interaction, stateful tool execution, and execution-grounded adjudication. Its current instantiation contains 22 attack families and 35 scenarios across seven domains, yielding more than 2,500 long-horizon trajectories with an average interaction length of 14.7 turns. Each trajectory is assigned one of five outcomes: Safe Completion, Correct Refusal, Unsafe Completion, Over-Refusal, or Indeterminate. Unlike fixed attack datasets, Blindspot is an extensible live-simulation framework in which attacks, scenarios, tools, policies, domains, and agent configurations can be added without redesigning the evaluation pipeline. We evaluate 13 proprietary and open-weight LLMs using eight metrics covering unsafe completion, appropriate refusal, benign utility, over-refusal, repeated-run robustness, and post-refusal failure. Preliminary results reveal substantial differences in safety-utility calibration across models and show that failures can emerge only after several initially safe interaction steps. These findings motivate treating agent safety as a trajectory-level property rather than a single-turn or binary success criterion.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.MA).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computational Engineering, Finance, and Science (cs.CE); Computation and Language (cs.CL); Machine Learning (cs.LG); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.16305 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:00:46.647Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).