Metraipolis › archive › dc_596c0cddde5d

ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures — arXiv:2609.16986

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16986
License (as archived)
CC0-1.0
Archived
2026-10-15T18:31:13.101Z
SHA-256 of the archived text
2d4ce1d90cd303e1975220a2ee1d55f957c2d9d105115b3ecbadd5e932896240
Stable URL of this page
https://metraipolis.com/documento/dc_596c0cddde5d

How to cite

ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures — arXiv:2609.16986 — arXiv (en), https://arxiv.org/abs/2609.16986 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_596c0cddde5d (Metraipolis archive dc_596c0cddde5d, sha256 2d4ce1d90cd303e1975220a2ee1d55f957c2d9d105115b3ecbadd5e932896240)
BibTeX
@misc{dc596c0cddde5d,
  title = {ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures — arXiv:2609.16986},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16986},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_596c0cddde5d; sha256 2d4ce1d90cd303e1975220a2ee1d55f957c2d9d105115b3ecbadd5e932896240},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16986. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-based multi-agent systems can fail even when communication succeeds because agents do not correctly track their peers' roles, knowledge, or intentions. We investigate whether such inter-agent misalignment cases, labelled FC2 in MAST-Data, can be converted into functional partner-state reasoning items. ToMAS applies four explicit convertibility criteria to diagnosed execution traces. A full conversion pass over 242 eligible non-AG2 training traces produced 39 CLEAN items. In an 18-trace reliability pilot, two annotators achieved 94.4% raw agreement and Cohen's kappa = 0.92. We then used the converted items as binary rewards in a small-scale GRPO feasibility experiment with Qwen2.5-1.5B. On a 28-item held-out Magentic GAIA diagnostic, every evaluated condition exceeded the ROUGE-L threshold on the same 2 of 28 items. Post-hoc adapter checks show why: under the learning rate used, the LoRA update remained numerically negligible (max abs Delta W about 7e-6), so all conditions decode identically to the untrained checkpoint. The experiment therefore does not show a training effect and cannot establish one; it reports an executable pipeline together with two limitations that any conclusive study must address: a provenance gap between the training and evaluation items, and lexical-overlap scoring. ToMAS provides a preliminary rubric and pipeline for converting diagnosed coordination failures into trainable partner-state reasoning items and identifies the requirements for a conclusive matched-domain evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.MA).
Categorie arXiv: Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.16986 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:00:46.630Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).