Metraipolis › archive › dc_218e428da31e

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources — arXiv:2610.19268

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19268
License (as archived)
CC0-1.0
Archived
2026-10-15T14:43:15.234Z
SHA-256 of the archived text
5dc880f1c92ab9a504b90bb4e5b879623284ae238835c1e49323f116910014d8
Stable URL of this page
https://metraipolis.com/documento/dc_218e428da31e

How to cite

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources — arXiv:2610.19268 — arXiv (en), https://arxiv.org/abs/2610.19268 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_218e428da31e (Metraipolis archive dc_218e428da31e, sha256 5dc880f1c92ab9a504b90bb4e5b879623284ae238835c1e49323f116910014d8)
BibTeX
@misc{dc218e428da31e,
  title = {Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources — arXiv:2610.19268},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19268},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_218e428da31e; sha256 5dc880f1c92ab9a504b90bb4e5b879623284ae238835c1e49323f116910014d8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19268. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what benchmark scores represent and where evaluation claims are supported.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.ET, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Emerging Technologies (cs.ET); Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2610.19268 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T14:00:53.776Z dalla pagina https://arxiv.org/list/cs.ET/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).