Metraipolis › archive › dc_ceebf9dfcff0

Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks — arXiv:2608.03340

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.03340
License (as archived): CC0-1.0
Archived: 2026-10-31T12:31:00.658Z
SHA-256 of the archived text: 30d85be93feb5033209097ac1d9e18255b3da7c16f5dff2e4d99357d38111820
Stable URL of this page: https://metraipolis.com/documento/dc_ceebf9dfcff0

How to cite

Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks — arXiv:2608.03340 — arXiv (en), https://arxiv.org/abs/2608.03340 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ceebf9dfcff0 (Metraipolis archive dc_ceebf9dfcff0, sha256 30d85be93feb5033209097ac1d9e18255b3da7c16f5dff2e4d99357d38111820)

BibTeX

@misc{dcceebf9dfcff0,
  title = {Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks — arXiv:2608.03340},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03340},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ceebf9dfcff0; sha256 30d85be93feb5033209097ac1d9e18255b3da7c16f5dff2e4d99357d38111820},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03340. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Predicting LLM's capabilities on real-world tasks is essential, yet the extent to which performance on commonsense benchmarks predicts downstream performance remains underspecified. To establish the practical usability of widely adopted commonsense benchmarks, we evaluate 23 models from six families on four established commonsense benchmarks, four reworked variants, three non-commonsense controls, and eight downstream tasks requiring implicit social, pragmatic, temporal, or physical reasoning. We compare model rankings, compute controlled correlations, and use leave-one-family-out cross-validation to assess the criterion validity of commonsense benchmarks. Our results show that revised benchmarks largely preserve original model rankings and do not improve downstream predictive power. Commonsense benchmarks show consistent cross-family predictive validity for only a narrow subset of downstream tasks, with smaller or metric-specific gains elsewhere. Overall, standardized commonsense benchmarks provide task-dependent rather than broad evidence of downstream commonsense competence.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.03340 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:17:37.752Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).