Metraipolis › archive › dc_e62ec509a465

Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary — arXiv:2608.04240

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04240
License (as archived)
CC0-1.0
Archived
2026-10-31T11:13:38.402Z
SHA-256 of the archived text
9535e4b1400433a4cc3979f78fc2cce23b1cbc53eb7f65d6d0d4652b56a09f9a
Stable URL of this page
https://metraipolis.com/documento/dc_e62ec509a465

How to cite

Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary — arXiv:2608.04240 — arXiv (en), https://arxiv.org/abs/2608.04240 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e62ec509a465 (Metraipolis archive dc_e62ec509a465, sha256 9535e4b1400433a4cc3979f78fc2cce23b1cbc53eb7f65d6d0d4652b56a09f9a)
BibTeX
@misc{dce62ec509a465,
  title = {Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary — arXiv:2608.04240},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04240},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e62ec509a465; sha256 9535e4b1400433a4cc3979f78fc2cce23b1cbc53eb7f65d6d0d4652b56a09f9a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04240. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.04240 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.490Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).