Metraipolis › archive › dc_ebc2e6a5a64b

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks — arXiv:2609.19182

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19182
License (as archived)
CC0-1.0
Archived
2026-10-10T03:19:28.412Z
SHA-256 of the archived text
6fae15a9c357d94294d85bd9101b9b4930d76749bc7d17a6e922b541b0a7cca7
Stable URL of this page
https://metraipolis.com/documento/dc_ebc2e6a5a64b

How to cite

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks — arXiv:2609.19182 — arXiv (en), https://arxiv.org/abs/2609.19182 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebc2e6a5a64b (Metraipolis archive dc_ebc2e6a5a64b, sha256 6fae15a9c357d94294d85bd9101b9b4930d76749bc7d17a6e922b541b0a7cca7)
BibTeX
@misc{dcebc2e6a5a64b,
  title = {What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks — arXiv:2609.19182},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19182},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebc2e6a5a64b; sha256 6fae15a9c357d94294d85bd9101b9b4930d76749bc7d17a6e922b541b0a7cca7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19182. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how evaluation requirements themselves are changing. The expanding variety of benchmarks offers another perspective: what researchers expect LLMs to do, and what they count as successful performance. We systematically map 14,767 papers introducing or updating evaluation resources from arXiv submissions between January 2022 and August 2026. Using staged screening and automated full-text coding, we examine changes in target systems and domains, evaluation materials and conditions, and scoring mechanisms. The collection shows growing emphasis on action, interaction, and professional applications, while established and newer design elements frequently coexist. Model participation also develops unevenly: LLM-based scoring grows within both agent and non-agent groups, whereas model-generated materials show no comparable sustained increase in recent cohorts. These findings illuminate how public research translates capability expectations into concrete tests and criteria for success. As AI participates in constructing tests, performing tasks, and judging responses, they also raise a question: does expanding evaluation provide more independent evidence, or risk reproducing the preferences and blind spots of its participating models?

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.19182 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:04:13.590Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).