Metraipolis › archive › dc_3c233e7908ec
Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM — arXiv:2611.02432
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02432
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:10:49.385Z
- SHA-256 of the archived text
8f6ce2667dc9ead884802233f23fca0d535b586ff5e2022e02ffaef061fc7606- Stable URL of this page
https://metraipolis.com/documento/dc_3c233e7908ec
How to cite
Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM — arXiv:2611.02432 — arXiv (en), https://arxiv.org/abs/2611.02432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3c233e7908ec (Metraipolis archive dc_3c233e7908ec, sha256 8f6ce2667dc9ead884802233f23fca0d535b586ff5e2022e02ffaef061fc7606)
BibTeX
@misc{dc3c233e7908ec,
title = {Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM — arXiv:2611.02432},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02432},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3c233e7908ec; sha256 8f6ce2667dc9ead884802233f23fca0d535b586ff5e2022e02ffaef061fc7606},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models with similar benchmark accuracy can succeed on different items. Overall scores do not show where their performance differs or which item properties might explain those differences. We demonstrate how the latent space item response model (LSIRM) can help evaluators investigate these patterns and their consequences for model comparison. We analyze 1,348 language models on 5,761 BBH items and compare ways of presenting and scoring answers across BBH, GPQA, MMLU-Pro, and MuSR. We examine LSIRM maps, which summarize differences in performance across models and items, alongside item properties and records of correct and incorrect responses. In BBH, nearby items on these maps often share the same correct answer, such as "Yes" or "No." Across the four benchmarks, this pattern is stronger when answers are scored using option labels, such as A, B, C, and D, than when using the text of the options. We then estimate each model's tendency to perform better or worse on items with particular correct answers and quantify how this tendency contributes to its score. This separates an estimated contribution associated with correct answers from the remaining differences in item performance. Two models' total numbers of correct answers differed by only 14 out of 5,761 items. Excluding the estimated answer-category contribution from their scores moved their ranks hundreds of places in opposite directions, while keeping all other estimates unchanged. The approach helps evaluators identify differences hidden by overall accuracy and understand how the choice of what to include in a score affects model rankings. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG); Methodology (stat.ME). Pagina del lavoro: https://arxiv.org/abs/2611.02432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:52.338Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).