Metraipolis › archive › dc_b8e006edef29

Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets — arXiv:2611.02639

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02639
License (as archived)
CC0-1.0
Archived
2026-11-03T11:07:48.889Z
SHA-256 of the archived text
b3a9fd069796c22fb09df720c68484ff0d25a4cae98ff342ab647df1c7eed84e
Stable URL of this page
https://metraipolis.com/documento/dc_b8e006edef29

How to cite

Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets — arXiv:2611.02639 — arXiv (en), https://arxiv.org/abs/2611.02639 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b8e006edef29 (Metraipolis archive dc_b8e006edef29, sha256 b3a9fd069796c22fb09df720c68484ff0d25a4cae98ff342ab647df1c7eed84e)
BibTeX
@misc{dcb8e006edef29,
  title = {Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets — arXiv:2611.02639},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02639},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b8e006edef29; sha256 b3a9fd069796c22fb09df720c68484ff0d25a4cae98ff342ab647df1c7eed84e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02639. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Overall and group-level averages, and ranking metrics such as the AUC, can hide sets of observations for which a prediction model systematically over- or under-predicts. We present an audit that asks where a model is miscalibrated (in which sets of similar observations it over- or under-predicts on average) and for whom (whether its average prediction error differs between demographic groups, such as by race or sex, within the same set). It groups observations into pockets using random-forest similarity, which weighs each feature by its importance for predicting the outcome, with a ridge-regression grouping alongside. Pockets are learned on one sample and tested on another, with simultaneous confidence intervals that allow for correlation within units such as hospitals; the audit needs only features, outcomes and fixed predictions. In 17 simulation scenarios with an error planted for one group, the combined audit located it better than the strongest alternative grouping in 15, with false-alarm rates near or below 5%; with a local error planted for everyone, it won in 12 of 15 scenarios at the smaller error size and lost when the error was unrelated to the outcome. In public synthetic data from the care-management algorithm studied by Obermeyer et al., it found no detectable overall error but under-prediction of illness in the sickest pockets and over-prediction elsewhere, and distinguished Black patients' concentration where the score under-predicted for everyone from a Black-White gap within pockets. For APACHE IVa, an exploratory analysis located its previously reported over-prediction of ICU mortality in a mostly ventilated pocket and a low-risk pocket, with none detected in a moderate-risk one. Two chest X-ray classifiers showed offsetting local errors behind no detectable overall error. Code: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Methodology (stat.ME); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2611.02639 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.115Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).