Metraipolis › archive › dc_417ed33ec141

AgentSpy: Making AI Agent Behavior Observable — arXiv:2610.06001

Source
arXiv
Original
https://arxiv.org/abs/2610.06001
License (as archived)
CC0-1.0
Archived
2026-10-06T05:27:20.627Z
SHA-256 of the archived text
3ed2928160baec0c6a628ce9caae6107e4ef86d3049b670d26fbacc30666ef91
Stable URL of this page
https://metraipolis.com/documento/dc_417ed33ec141

How to cite

AgentSpy: Making AI Agent Behavior Observable — arXiv:2610.06001 — arXiv, https://arxiv.org/abs/2610.06001 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_417ed33ec141 (Metraipolis archive dc_417ed33ec141, sha256 3ed2928160baec0c6a628ce9caae6107e4ef86d3049b670d26fbacc30666ef91)
BibTeX
@misc{dc417ed33ec141,
  title = {AgentSpy: Making AI Agent Behavior Observable — arXiv:2610.06001},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06001},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_417ed33ec141; sha256 3ed2928160baec0c6a628ce9caae6107e4ef86d3049b670d26fbacc30666ef91},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06001. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI agents built on large language models (LLMs) run shell commands, read and write files, and reach the network, typically with their user's privileges. However, what an agent does during an execution is difficult to understand: tests assert on the result, and the agent's trajectory records only what the agent reports about itself, which may omit behavior executed by its subprocesses. We present AgentSpy, an approach that observes an agent from outside the agent. AgentSpy runs the agent in an isolated environment, configured by a declarative specification, and records the system calls and network traffic of the agent and of every process it executes. Based on this monitoring, AgentSpy supports two families of analyses: conformance analyses, which measure obligations, i.e., what an agent execution should do, and safety analyses, which check prohibitions, i.e., what an agent execution must never do. We instantiate one analysis of each family. The reliability analysis uses rules to summarize each run by the environment resources the agent uses: the commands it executed, the files it accessed, and the hosts it contacted. The security analysis applies deterministic rules to the system calls of an execution. For reliability, we evaluated AgentSpy on 77 tasks with the codex harness and three recent LLMs, executing each task three times. Sets of repeated runs of the same task are more similar than sets that include runs of another task in 92.2% of the comparisons. Among tasks for which all three runs pass outcome-based tests, the agent performs task-unrelated activities in 18% of the cases, reads the grading files in 7%, and does not use the developers' guidance in 17%. For security, the generic rules of AgentSpy detect four of five attack categories we considered, with no false positives across 50 runs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.06001 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.101Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).