Metraipolis › archive › dc_66fc3290541e

MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents — arXiv:2611.07689

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07689
License (as archived): CC0-1.0
Archived: 2026-11-06T14:41:43.001Z
SHA-256 of the archived text: d964d8c549348a87e69c3588dcad87ad09f6ec3d041ba7449778ad41b3517775
Stable URL of this page: https://metraipolis.com/documento/dc_66fc3290541e

How to cite

MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents — arXiv:2611.07689 — arXiv (en), https://arxiv.org/abs/2611.07689 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_66fc3290541e (Metraipolis archive dc_66fc3290541e, sha256 d964d8c549348a87e69c3588dcad87ad09f6ec3d041ba7449778ad41b3517775)

BibTeX

@misc{dc66fc3290541e,
  title = {MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents — arXiv:2611.07689},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07689},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_66fc3290541e; sha256 d964d8c549348a87e69c3588dcad87ad09f6ec3d041ba7449778ad41b3517775},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07689. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model agents are increasingly deployed through the Model Context Protocol (MCP), which allows them to discover and invoke tools across multiple servers at runtime. Existing MCP benchmarks primarily measure task completion; where they diagnose failures, they do so after execution rather than building specific reasoning, execution, and memory failure risks into task construction. We introduce MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory. MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers. Each task is designed with a structural trigger that makes one of four subtypes the dominant failure risk: Branch Collapse, Unreachable Goal, Tool Misuse, or Context Forgetting. This design enables subtype-level analysis rather than aggregate task success alone. We evaluate fourteen frontier LLM agents from eight vendors on 5{,}012 trajectories. Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better). Failures concentrate on Unreachable Goal tasks, where the average score drops to 0.458 and 23.2\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals. Subtype rankings diverge sharply: a model's rank shifts by 6.9 of 13 positions on average across subtypes, and Branch Collapse is uncorrelated with Unreachable Goal across models ($r = -0.12$). MCPHallu is released at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.07689 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:42.644Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).