Metraipolis › archive › dc_bf50f19ce02e

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models — arXiv:2611.00460

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00460
License (as archived)
CC0-1.0
Archived
2026-11-03T10:51:37.507Z
SHA-256 of the archived text
199e68f2163490874e9c982422155218da50633370c0eaf9da1f62c208fb2b20
Stable URL of this page
https://metraipolis.com/documento/dc_bf50f19ce02e

How to cite

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models — arXiv:2611.00460 — arXiv (en), https://arxiv.org/abs/2611.00460 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf50f19ce02e (Metraipolis archive dc_bf50f19ce02e, sha256 199e68f2163490874e9c982422155218da50633370c0eaf9da1f62c208fb2b20)
BibTeX
@misc{dcbf50f19ce02e,
  title = {From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models — arXiv:2611.00460},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00460},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf50f19ce02e; sha256 199e68f2163490874e9c982422155218da50633370c0eaf9da1f62c208fb2b20},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00460. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical competence, or are instead affected by the format of the questions being asked (e.g., multiple-choice versus open-ended), prior exposure to the benchmarks, and the difficulty of the questions. These benchmarks are also public and widely used online, which may be resulting in LLM memorization. Models may perform better on questions that they have seen before. It is also not evident whether smaller models, which are more locally deployable in HIPAA-compliant hospital settings, have achieved human performance. In this study, we evaluate fourteen current models from six leading providers, one smaller and one larger model from each provider with two additional open-weight models, on 3,076 questions from four medical benchmarks (MedQA-USMLE, MedMCQA, PubMedQA, and MedBullets) under standard multiple choice, the same questions with the answer choices withheld, and reconstruction of a hidden question from its choices alone. In the standard multiple-choice setting, the strongest models exceed the human-expert baseline on MedQA-USMLE and PubMedQA, but the smaller models trail their larger models by 8.7 percentage points and only two of the six pass the human-expert bar on the licensing examination. Withholding the answer choices, however, makes every model struggle: mean accuracy falls from 82.3% to 50.9%, and the two largest drops belong to the medically specialised models. On memorization, the strongest models reconstruct one question in four from the answer choices alone, yet accuracy holds on the questions they cannot reconstruct. We further weigh accuracy against inference cost and test whether ensembling several models improves medical question answering.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.00460 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.244Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).