Metraipolis › archive › dc_e8bc8e68cf14

Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes — arXiv:2610.40318

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40318
License (as archived)
CC0-1.0
Archived
2026-10-31T06:14:52.548Z
SHA-256 of the archived text
4fc61b89a5bffb776360064a53b8702adb96d0a33aee03d6cddb9e80ae2473cd
Stable URL of this page
https://metraipolis.com/documento/dc_e8bc8e68cf14

How to cite

Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes — arXiv:2610.40318 — arXiv (en), https://arxiv.org/abs/2610.40318 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e8bc8e68cf14 (Metraipolis archive dc_e8bc8e68cf14, sha256 4fc61b89a5bffb776360064a53b8702adb96d0a33aee03d6cddb9e80ae2473cd)
BibTeX
@misc{dce8bc8e68cf14,
  title = {Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes — arXiv:2610.40318},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40318},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e8bc8e68cf14; sha256 4fc61b89a5bffb776360064a53b8702adb96d0a33aee03d6cddb9e80ae2473cd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40318. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Objectives: To evaluate the accuracy and calibration of Decision and System One models on rheumatology examination questions and diagnostic vignettes. Methods: Two benchmarks were analysed: 172 rheumatology questions from Spanish specialty-training examinations (expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench with 5-18 candidate diagnoses, 52 of which included the reference diagnosis. Four System One models (Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna (OpenAI Decisions API) were compared with GPT-6 Sol and two local generative models. The primary outcome was accuracy; secondary outcomes included calibration, discrimination, accuracy of answers with probability >=0.90 and language differences. Results: In RheumaMIR, Jev's accuracy was 93.0% in Spanish and 91.9% in English, below GPT-6 Sol (99.4%; Holm-adjusted P<=0.005); GPT-6 Luna reached 97.7%, Clef 91.9-93.0% and Microsoft-Decision-1 86.0-87.8%, without detectable language differences. Jev's answers with probability >=0.90 (76-77% of answers) were 98.5% correct. In RheumBench, accuracy was lower: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0% and Microsoft-Decision-1 40.4% (each System One model below GPT-6 Sol, Holm-adjusted P<=0.045; chance 9.6%). Jev became overconfident (mean probability 0.659; over/underconfidence index +0.159), and its answers with probability >=0.90 (25% of cases) were 84.6% correct, against 95.5% for GPT-6 Luna (42%). Laya performed near chance in both benchmarks. Conclusion: The best-performing System One models were highly accurate on examination questions but less accurate on diagnostic vignettes, remaining below GPT-6 Sol. Jev's high-confidence predictions were less reliable on vignettes, highlighting the need for task-specific evaluation and calibration.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.40318 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.765Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).