Metraipolis › archive › dc_775498ade401

Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage — arXiv:2610.01963

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01963
License (as archived)
CC0-1.0
Archived
2026-10-06T12:06:34.257Z
SHA-256 of the archived text
abb9b0ebae3e4b72bd870231057dc677c87939d8e66cedee793cde28f330522a
Stable URL of this page
https://metraipolis.com/documento/dc_775498ade401

How to cite

Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage — arXiv:2610.01963 — arXiv (en), https://arxiv.org/abs/2610.01963 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_775498ade401 (Metraipolis archive dc_775498ade401, sha256 abb9b0ebae3e4b72bd870231057dc677c87939d8e66cedee793cde28f330522a)
BibTeX
@misc{dc775498ade401,
  title = {Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage — arXiv:2610.01963},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01963},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_775498ade401; sha256 abb9b0ebae3e4b72bd870231057dc677c87939d8e66cedee793cde28f330522a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01963. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Emergency department (ED) triage is a high-stakes prioritization task in which demographic, socioeconomic, and system-context information may improperly influence acuity assignment. Although open-source large language models (LLMs) are increasingly considered for local and privacy-preserving clinical decision support, it remains unclear how counterfactual bias varies across model families, sizes, medical-domain models, and domain-adapted models. We present a comparative counterfactual audit of ten open-source LLMs for pediatric Emergency Severity Index (ESI) prediction. Starting from real and handbook-style clinical vignettes, we construct paired counterfactual variants that change only one injected demographic, socioeconomic, healthcare-access, behavioral, social, or system-context variable while holding the clinical presentation fixed. Models include Qwen2.5-7B, Qwen2.5-14B-Instruct, a QLoRA fine-tuned Qwen2.5-7B, MedGemma variants, MedLLaMA2-7B, GPT-OSS-20B, and GPT-OSS-120B. We measure any counterfactual shift, undertriage, overtriage, shifts greater than one ESI level, mean shift, and mean absolute shift. Counterfactual sensitivity varied substantially and did not consistently decrease with larger model size or medical-domain pretraining. The fine-tuned Qwen2.5-7B showed the lowest overall sensitivity, with a 5.27% any-shift rate and mean absolute shift of 0.0534, versus 16.02% and 0.1706 for the base model. Several larger or medical-domain models showed more significant shifts. Stratified and correlation analyses further revealed clinically important directionality and shared failure patterns hidden by aggregate rates. These findings support counterfactual auditing as a lightweight, clinically interpretable framework for comparing fairness risks in open-source LLMs before clinical deployment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.01963 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.986Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).