Metraipolis › archive › dc_e28210dc3173

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals — arXiv:2609.34228

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34228
License (as archived)
CC0-1.0
Archived
2026-10-07T22:38:06.031Z
SHA-256 of the archived text
32cb59258404078ce1df7aa97488d57ad0f9cab14434b2445e0e632628e66df8
Stable URL of this page
https://metraipolis.com/documento/dc_e28210dc3173

How to cite

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals — arXiv:2609.34228 — arXiv (en), https://arxiv.org/abs/2609.34228 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e28210dc3173 (Metraipolis archive dc_e28210dc3173, sha256 32cb59258404078ce1df7aa97488d57ad0f9cab14434b2445e0e632628e66df8)
BibTeX
@misc{dce28210dc3173,
  title = {SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals — arXiv:2609.34228},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34228},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e28210dc3173; sha256 32cb59258404078ce1df7aa97488d57ad0f9cab14434b2445e0e632628e66df8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34228. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating statistical discovery by large language model (LLM) agents requires verifiable analytical ground truth. Establishing such ground truth for real-world datasets is costly, and prior knowledge of public datasets can influence agent responses. We introduce SLEUTHBENCH, a benchmark that addresses both problems by injecting controlled data-quality problems and feature effects into public tabular datasets: the injected pattern determines the answer, so reference answers are computed automatically and memorized knowledge of the original table is insufficient, while the table keeps its background structure. The injected patterns are modeled on phenomena reported in real data analyses. The benchmark defines 17 question templates in two families: data-quality questions and feature-contribution questions. We evaluate six state-of-the-art LLMs that analyze the data using a Python coding tool, on data-science and business phrasings of 70 validated dataset-template combinations, yielding 1680 graded responses in total. The models detect data-quality problems reliably (83.8% accuracy) but recover feature contributions poorly (41.9%). Finding how features shape the target requires searching over both candidate variables and analytical procedures. To address this issue, we propose the Empirical Layer, a set of precomputed statistical artifacts comprising summaries, fitted feature and interaction effects, and dataset descriptions, which exposes candidate patterns for direct inspection. Access to these artifacts raises feature-contribution accuracy from 41.9% to 68.0%.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.34228 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.139Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).