Metraipolis › archive › dc_cc250cd0cc26
Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting — arXiv:2609.26288
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.26288
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T15:07:55.896Z
- SHA-256 of the archived text
f6a04e8540650715eca75985dc23424a1c91d1bdc68363f397e31eff2904207f- Stable URL of this page
https://metraipolis.com/documento/dc_cc250cd0cc26
How to cite
Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting — arXiv:2609.26288 — arXiv (en), https://arxiv.org/abs/2609.26288 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cc250cd0cc26 (Metraipolis archive dc_cc250cd0cc26, sha256 f6a04e8540650715eca75985dc23424a1c91d1bdc68363f397e31eff2904207f)
BibTeX
@misc{dccc250cd0cc26,
title = {Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting — arXiv:2609.26288},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.26288},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cc250cd0cc26; sha256 f6a04e8540650715eca75985dc23424a1c91d1bdc68363f397e31eff2904207f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26288. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Comparative studies of predictive models often end by ranking candidate models, yet these rankings depend on evaluation protocols whose influence is rarely treated as a source of uncertainty. We formalize this problem as protocol-induced ranking uncertainty and introduce a framework that compares ranking displacement caused by switching protocols with displacement produced by conventional choices within a fixed protocol. We quantify these effects using the Protocol Sensitivity Score (PSS) and a full-refit intraprotocol reference. We validate the framework in a large-scale sequential prediction study of daily PM10. Static-split and rolling-origin evaluation are compared across 425 European background stations and 365 US EPA monitors. Switching protocols produces mean PSS values of 0.801 and 0.772 and changes the selected model at 35.3% and 31.5% of stations, respectively. In Europe, intraprotocol perturbations with identical scored targets produce PSS values of 0.072 and 0.230, with winner-swap rates of 0.8% and 4.9%. Between-protocol displacement is therefore substantially larger than the selected within-protocol references. Expanding the candidate set from three to nine models increases the between-protocol winner-swap rate to 60.2% in Europe. The pattern also persists under a frozen protocol applied to held-out background and non-background stations. These results show that model-selection conclusions can depend materially on legitimate evaluation choices. We recommend reporting ranking stability under a small set of defensible intraprotocol perturbations alongside claims of model superiority. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.26288 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T14:37:29.481Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).