Metraipolis › archive › dc_f1d90645bb5d

The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs — arXiv:2608.27750

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27750
License (as archived)
CC0-1.0
Archived
2026-10-17T10:56:41.098Z
SHA-256 of the archived text
e0324f97c2e54deab330887c45e0e7caea8a1e43b2f34ecb7432bcd06968dffe
Stable URL of this page
https://metraipolis.com/documento/dc_f1d90645bb5d

How to cite

The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs — arXiv:2608.27750 — arXiv (en), https://arxiv.org/abs/2608.27750 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f1d90645bb5d (Metraipolis archive dc_f1d90645bb5d, sha256 e0324f97c2e54deab330887c45e0e7caea8a1e43b2f34ecb7432bcd06968dffe)
BibTeX
@misc{dcf1d90645bb5d,
  title = {The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs — arXiv:2608.27750},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27750},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f1d90645bb5d; sha256 e0324f97c2e54deab330887c45e0e7caea8a1e43b2f34ecb7432bcd06968dffe},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27750. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The hidden states of large language models (LLMs) are known to capture rich information relating to model knowledge and behavior that can be hard to extract from examination of input and output alone. As LLM-based systems increasingly interface with the external world, one area of concern is detecting incorrect or improper use of tools. Motivated by this, we study the effectiveness of using linear probes to detect incorrect tool-calls, measuring probe efficacy across 18 tool-calling LLMs evaluated on the Berkeley Function Calling Leaderboard. Overall, we find that probing is an effective means to catch a range of different tool-calling errors, including errors arising from using an argument that has the wrong value but the correct type, which might not be recorded by standard logging frameworks. Important factors in success include model size, probing layer, and model post-training type. We also show that probes are capable of generalizing to novel types of errors, which is critical in real world deployments.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.27750 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:47:33.152Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).