Metraipolis › archive › dc_fde96dd77cd7

MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection — arXiv:2611.05340

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.05340
License (as archived): CC0-1.0
Archived: 2026-11-04T12:31:21.367Z
SHA-256 of the archived text: 223badaacf942a6d926a9e2790bdadda1e940ca40125d8ce877b2bcb3400382c
Stable URL of this page: https://metraipolis.com/documento/dc_fde96dd77cd7

How to cite

MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection — arXiv:2611.05340 — arXiv (en), https://arxiv.org/abs/2611.05340 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fde96dd77cd7 (Metraipolis archive dc_fde96dd77cd7, sha256 223badaacf942a6d926a9e2790bdadda1e940ca40125d8ce877b2bcb3400382c)

BibTeX

@misc{dcfde96dd77cd7,
  title = {MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection — arXiv:2611.05340},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05340},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fde96dd77cd7; sha256 223badaacf942a6d926a9e2790bdadda1e940ca40125d8ce877b2bcb3400382c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05340. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it. MedPhysBench has ten seeded generators of calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, up to clinical tasks such as a TG-51 calibration worksheet. Reference answers come from code implementing the governing protocols, and a grader converts units and applies tolerance bands. From the same inputs, separate checking items test error detection, showing a colleague's worksheet or only the colleague's result, half correct and half with one planted error. On a held-out test split of 1,200 items (half for two models), 12 models scored 33.3 to 100.0% on calculation. Of the nine answers graded incorrect for the four best (99.3-100.0%), five contained model errors and four followed NCRP Report 151 on points our items misstate or leave open. Below them, most models did worst on level-3 clinical tasks, and Claude Sonnet 5.5 omitted the inverse-square correction in 19 of 20 isocentric MU items. Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms; for four of the other seven, checking only the result raised false alarms significantly. In an exploratory targeted test with 20 public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet, and three weaker models accepted 16 to 18 of 20 planted worksheets, one of them while applying the correction in most of its own calculations. Claude Opus 5.5 accepted 6 of 20 wrong results, although it applied the correction in all 20 of its own. We report the effect of two problems in our shielding items, found after the runs, and release the code, per-item outcomes and targeted-test responses.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Medical Physics (physics.med-ph) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05340 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:26:47.331Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).