Metraipolis › archive › dc_0928ccf45716
Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset — arXiv:2609.19019
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.19019
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T15:39:22.611Z
- SHA-256 of the archived text
c0b1f18bc2b9d17ab42b598349557c1bf5a47c651c2e5bf4a36a6464182bbb53- Stable URL of this page
https://metraipolis.com/documento/dc_0928ccf45716
How to cite
Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset — arXiv:2609.19019 — arXiv (en), https://arxiv.org/abs/2609.19019 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0928ccf45716 (Metraipolis archive dc_0928ccf45716, sha256 c0b1f18bc2b9d17ab42b598349557c1bf5a47c651c2e5bf4a36a6464182bbb53)
BibTeX
@misc{dc0928ccf45716,
title = {Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset — arXiv:2609.19019},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.19019},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0928ccf45716; sha256 c0b1f18bc2b9d17ab42b598349557c1bf5a47c651c2e5bf4a36a6464182bbb53},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19019. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Model validation is routinely used to quantify and correct for overfitting in performance estimates, but it does not fully address the stability of models developed from slightly different training samples. We investigated whether stable discrimination implies stable individual predictions and compared logistic regression (LR) with artificial neural networks (ANNs). Using data from 40,830 participants in the GUSTO-I trial, with 30-day mortality as the outcome, we created six sample-size scenarios with varying events per variable (EPV). In each scenario, LR and ANN models were developed using eight predictors, and 200 bootstrap resamples were used to assess stability in discrimination, mean absolute prediction error (MAPE), and the classification instability index (CII). For LR, discrimination stabilized at an EPV of 44.50, whereas MAPE and CII did not stabilize until an EPV of 89.12. ANN discrimination was more volatile, with optimism stabilizing only at an EPV of 178.25, and ANN predictions were severely miscalibrated at an EPV of 4.50. Overall, discrimination stabilized earlier than individual predictions for both approaches, with greater instability and larger sample-size requirements for ANN. Model developers should therefore assess and report prediction-level stability, because discrimination metrics alone may not reflect the reliability of individual predictions. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-17 in stat.ME). Categorie arXiv: Methodology (stat.ME). Pagina del lavoro: https://arxiv.org/abs/2609.19019 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T15:05:19.050Z dalla pagina https://arxiv.org/catchup/stat.ME/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).