Metraipolis › archive › dc_8a4e6a31c1fd

MLLMs Fail to Refuse when Using Tools Agentically — arXiv:2610.03938

Source
arXiv
Original
https://arxiv.org/abs/2610.03938
License (as archived)
CC0-1.0
Archived
2026-10-06T05:24:11.009Z
SHA-256 of the archived text
7616d043888111096d9db4a0e98ef72d1135f0503754497942b816c3353dee9f
Stable URL of this page
https://metraipolis.com/documento/dc_8a4e6a31c1fd

How to cite

MLLMs Fail to Refuse when Using Tools Agentically — arXiv:2610.03938 — arXiv, https://arxiv.org/abs/2610.03938 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8a4e6a31c1fd (Metraipolis archive dc_8a4e6a31c1fd, sha256 7616d043888111096d9db4a0e98ef72d1135f0503754497942b816c3353dee9f)
BibTeX
@misc{dc8a4e6a31c1fd,
  title = {MLLMs Fail to Refuse when Using Tools Agentically — arXiv:2610.03938},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03938},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8a4e6a31c1fd; sha256 7616d043888111096d9db4a0e98ef72d1135f0503754497942b816c3353dee9f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03938. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic multimodal large language models (MLLMs) have recently pushed the frontier of visual reasoning by calling tools such as zooming and tagging. Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests. Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%. Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.03938 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.407Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).