Metraipolis › archive › dc_1d4eb91c58ef
Condence, Calibration, and Abstention in Small Open-Weight Language Models on Psychiatric Knowledge Questions: A Cross-Domain Replication (2026) · DOI 10.21203/rs.3.rs-11012222/v1
- Source
- Metraipolis archive
- Original
- https://doi.org/10.21203/rs.3.rs-11012222/v1
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-08T01:20:03.402Z
- SHA-256 of the archived text
9eccea1244b8a9d0347d7c7d9acaaee9090b5db2c07f1dddfd2d79a99ab46cd3- Stable URL of this page
https://metraipolis.com/documento/dc_1d4eb91c58ef
How to cite
Condence, Calibration, and Abstention in Small Open-Weight Language Models on Psychiatric Knowledge Questions: A Cross-Domain Replication (2026) · DOI 10.21203/rs.3.rs-11012222/v1 — Metraipolis archive, https://doi.org/10.21203/rs.3.rs-11012222/v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_1d4eb91c58ef (Metraipolis archive dc_1d4eb91c58ef, sha256 9eccea1244b8a9d0347d7c7d9acaaee9090b5db2c07f1dddfd2d79a99ab46cd3)
BibTeX
@misc{dc1d4eb91c58ef,
title = {Condence, Calibration, and Abstention in Small Open-Weight Language Models on Psychiatric Knowledge Questions: A Cross-Domain Replication (2026) · DOI 10.21203/rs.3.rs-11012222/v1},
howpublished = {Metraipolis archive},
url = {https://doi.org/10.21203/rs.3.rs-11012222/v1},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_1d4eb91c58ef; sha256 9eccea1244b8a9d0347d7c7d9acaaee9090b5db2c07f1dddfd2d79a99ab46cd3},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.21203/rs.3.rs-11012222/v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content). Titolo: Condence, Calibration, and Abstention in Small Open-Weight Language Models on Psychiatric Knowledge Questions: A Cross-Domain Replication Autori: Kunal Dhanda Pubblicato: 2026-09-15 · editore: Springer Science and Business Media LLC DOI: https://doi.org/10.21203/rs.3.rs-11012222/v1 Licenza dell'articolo: CC-BY-4.0 Abstract (testo degli autori, CC-BY-4.0): Abstract Small, locally deployable language models are increasingly proposed for psychiatric clinical decision support on privacy and compute grounds; two contemporaneous 2026 systems (ClinMPO; PKFAR) report accuracy or reasoning-quality gains from augmenting such models, and a 120-study taxonomy of LLMs in mental healthcare lists calibration as an evaluated dimension in none of the studies it covers. A separate, contemporaneous study (Boie et al., 2026) evaluates verbalized-condence cali- bration across 20 medical specialties using six proprietary frontier models on MedMCQA, and a third (Bentegeac et al., 2025) shows across several medical QA benchmarks that verbalized condence is a weak predictor of correctness (AUROC 0.520.68) compared to token probability (AUROC 0.710.87). We narrow the open question this leaves for small, open-weight models specically to psychiatry, using the identical four-model ladder (1.2B9.2B parameters, Ollama, fully local) and harness as a companion general-medicine pilot, on 150 psychiatric-knowledge multiple-choice items (MedMCQA, subject = Psychiatry). Accuracy is higher than on the general-medicine benchmark (57.375.3% vs. 47.763.2%, though at n = 150 the per-model standard error is ±3.54.0 points), and the accuracy ranking partially reorders most notably the 4.3B model falls from 3rd of 4 in general medicine to last in psychiatry. Verbalized condence remains poorly informative about correctness as a ranking signal (AUROC 0.4900.645; one model, the 4.3B, is statistically indistin- guishable from chance) and severely miscalibrated in the aggregate sense (ECE 0.1480.378, well above the 0.0410.141 range Boie et al. report for proprietary frontier models on the same under- lying benchmark family). A naive, unpaired accuracy on answered items framing suggested that oering an abstention option modestly beneted three of the four models. A same-item paired analysis (McNemar's test, bootstrap condence intervals on paired accuracy, and an explicit A→B outcome-transition table) overturns most of that: only the smallest model (1.2B) shows a paired eect distinguishable from noise, and it is negative (−6.7 points, 95% CI [−12.7,−0.7]) repli- cating, point estimate for point estimate, the same model's paired eect on the general-medicine benchmark (−6.7 points, 95% CI [−10.6,−2.5], n = 480). The model with the best condence- ranking signal of the four (7.6B, AUROC 0.645) shows the worst-targeted self-chosen abstentions 8 of its 12 abstentions forfeited an item it would otherwise have answered correctly and its paired eect reverses from a naive +0.8 to a signicant −5.3 points (95% CI [−9.3,−1.3]). We 1read the surviving evidence as: miscalibration and weak error-detection are general properties of this model family; accuracy ranking is domain-dependent and should not be assumed to transfer; and a prompted abstention option is not a free safety improvement for any of the four models once evaluated with paired, coverage-aware statis... Fonte: https://api.crossref.org/works/10.21203%2Frs.3.rs-11012222%2Fv1 — metadati Crossref, consultati il 2026-10-07. Interrogazione: «small language models».