Metraipolis › archive › dc_fd1c47ff1e4d
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model — arXiv:2608.22295
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.22295
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T07:44:45.896Z
- SHA-256 of the archived text
e0d139656caf0f780d1f36125ea4d2d7abe0c980027aeb4370e42d80278c6c8c- Stable URL of this page
https://metraipolis.com/documento/dc_fd1c47ff1e4d
How to cite
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model — arXiv:2608.22295 — arXiv (en), https://arxiv.org/abs/2608.22295 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd1c47ff1e4d (Metraipolis archive dc_fd1c47ff1e4d, sha256 e0d139656caf0f780d1f36125ea4d2d7abe0c980027aeb4370e42d80278c6c8c)
BibTeX
@misc{dcfd1c47ff1e4d,
title = {LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model — arXiv:2608.22295},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.22295},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd1c47ff1e4d; sha256 e0d139656caf0f780d1f36125ea4d2d7abe0c980027aeb4370e42d80278c6c8c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22295. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluation of large language models (LLMs) increasingly requires predicting how a model will perform on new questions or tasks before collecting large amounts of new annotations. This problem is challenging because question difficulty, scenario, and underlying capability demands can vary substantially. Simple retrospective averages may confound model ability with item characteristics. In this paper, we study a model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions. The framework represents LLMs through latent capability profiles while using question content to inform item characteristics, allowing information to transfer beyond previously observed items. Empirically, we find that for within-scenario evaluation, incorporating question embeddings improves prediction relative to model-free baselines, and that multidimensional latent structure provides a richer description of capability variation than unidimensional alternatives. At the same time, our results reveal an important limitation that the generalizability does not necessarily translate into reliable prediction under cross-scenario shift. These findings suggest that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation, while also highlighting cross-scenario generalization as a central open challenge. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.22295 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T07:37:52.869Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).