Metraipolis › archive › dc_ab93cd96fb9e
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation — arXiv:2608.29887
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29887
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:07:13.568Z
- SHA-256 of the archived text
bb6ce8adbe6ad66c4ae73a87e5b332d793ac861fb32cb6b103ef9238ca241b64- Stable URL of this page
https://metraipolis.com/documento/dc_ab93cd96fb9e
How to cite
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation — arXiv:2608.29887 — arXiv (en), https://arxiv.org/abs/2608.29887 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ab93cd96fb9e (Metraipolis archive dc_ab93cd96fb9e, sha256 bb6ce8adbe6ad66c4ae73a87e5b332d793ac861fb32cb6b103ef9238ca241b64)
BibTeX
@misc{dcab93cd96fb9e,
title = {Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation — arXiv:2608.29887},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29887},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ab93cd96fb9e; sha256 bb6ce8adbe6ad66c4ae73a87e5b332d793ac861fb32cb6b103ef9238ca241b64},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29887. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.29887 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:47:52.748Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).