Metraipolis › archive › dc_fde2ecde4c5a

Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components — arXiv:2611.03337

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03337
License (as archived)
CC0-1.0
Archived
2026-11-03T11:30:27.385Z
SHA-256 of the archived text
c0e6f4dbe82378cdad98f15bf103787980abab5cd05b9cbdf43da8b70d120eb6
Stable URL of this page
https://metraipolis.com/documento/dc_fde2ecde4c5a

How to cite

Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components — arXiv:2611.03337 — arXiv (en), https://arxiv.org/abs/2611.03337 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fde2ecde4c5a (Metraipolis archive dc_fde2ecde4c5a, sha256 c0e6f4dbe82378cdad98f15bf103787980abab5cd05b9cbdf43da8b70d120eb6)
BibTeX
@misc{dcfde2ecde4c5a,
  title = {Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components — arXiv:2611.03337},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03337},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fde2ecde4c5a; sha256 c0e6f4dbe82378cdad98f15bf103787980abab5cd05b9cbdf43da8b70d120eb6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03337. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

There has been growing emphasis on acknowledging, quantifying, and communicating the uncertainty inherent in the output of clinical prediction models. However, allowing for model uncertainty when assessing the performance of an existing model in a new sample (i.e., external validation) has not been sufficiently explored. When validating a model that generates uncertain predictions, the uncertainty of a given performance metric is affected by both model uncertainty and the finiteness of the validation sample (validation uncertainty). For a given performance metric, we show that the total uncertainty of the performance estimate can be decomposed into the uncertainty due to the model and the uncertainty due to the validation. Also, we propose $M^2$, the fraction of the total variance due to model uncertainty, and $V^2$, the fraction of the total variance due to validation uncertainty. These measures serve as complementary scalar indices that can be implemented for common performance metrics and visualizations such as calibration plots using simple Monte Carlo algorithms. Critically, our case study shows that with the same model and the same validation sample, the contribution of uncertainty sources is highly variable across performance metrics ($M^2$ varying from 12% for the Brier score to 78% for the calibration slope). This indicates that stakeholders who prioritize different aspects of performance (e.g., calibration versus clinical utility) might arrive at different decisions on whether future efforts should be directed towards creating a more reliable model or procuring a larger validation sample. In conclusion, external validation studies can readily be amended to incorporate model uncertainty, and we recommend that such studies report total variance as well as the relative share of variance components for all performance metrics.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv stat.ME, nuovi lavori del giorno).
Categorie arXiv: Methodology (stat.ME).
Pagina del lavoro: https://arxiv.org/abs/2611.03337 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:03:37.237Z dalla pagina https://arxiv.org/list/stat.ME/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).