Metraipolis › archive › dc_867a4dfec7da
What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty — arXiv:2610.39373
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.39373
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:13:13.008Z
- SHA-256 of the archived text
cc71afab5287989de4add6c7b2d6012a2b1e4de48aeda99e3bef9173a23840f8- Stable URL of this page
https://metraipolis.com/documento/dc_867a4dfec7da
How to cite
What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty — arXiv:2610.39373 — arXiv (en), https://arxiv.org/abs/2610.39373 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_867a4dfec7da (Metraipolis archive dc_867a4dfec7da, sha256 cc71afab5287989de4add6c7b2d6012a2b1e4de48aeda99e3bef9173a23840f8)
BibTeX
@misc{dc867a4dfec7da,
title = {What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty — arXiv:2610.39373},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.39373},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_867a4dfec7da; sha256 cc71afab5287989de4add6c7b2d6012a2b1e4de48aeda99e3bef9173a23840f8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39373. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Research on medical item difficulty has largely focused on prediction rather than on identifying interpretable patterns associated with empirical difficulty. This study used empirical difficulty labels to guide large language models (LLMs) in inducing difficulty rubrics and examined how rubric guidance affected LLM difficulty judgments. Using training items and their empirical difficulty estimates from the United States Medical Licensing Examination (USMLE) and the Physician Assistant National Certifying Examination (PANCE), Claude Opus 5 induced separate rubrics for absolute and pairwise difficulty judgment. None of the induced rubrics treated surface features, such as stem length, option count, or images, as major sources of difficulty. Several patterns emerged across both examinations, including cue convergence, misleading cues, and option competition, although their specific forms differed. Retrieval of knowledge that could not be derived from the stem was particularly prominent in the PANCE rubrics, whereas the USMLE rubrics placed greater emphasis on integrating multiple clinical cues. We compared commercial and open-weight LLMs on held-out items under unguided and rubric-guided conditions. Rubric guidance changed how models used the difficulty scale but did not consistently improve performance. In absolute judgments, guidance encouraged broader use of the scale and generally improved coarse-grained agreement, without consistently improving rank ordering. In pairwise comparisons, its effects varied across models and examinations, and judgments became more sensitive to presentation order. These findings suggest that the value of label-guided rubric induction may lie more in generating interpretable, testable hypotheses from empirical difficulty data than in reliably improving difficulty judgments. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.39373 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:08.711Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).