Metraipolis › archive › dc_7db7fe592799

Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent — arXiv:2609.03221

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03221
License (as archived)
CC0-1.0
Archived
2026-10-07T12:43:13.651Z
SHA-256 of the archived text
0bdfbaeae713c1833b3cd450857933b6e6f6ba0d0016d2cb55e16c3f0ec2fe6d
Stable URL of this page
https://metraipolis.com/documento/dc_7db7fe592799

How to cite

Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent — arXiv:2609.03221 — arXiv (en), https://arxiv.org/abs/2609.03221 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7db7fe592799 (Metraipolis archive dc_7db7fe592799, sha256 0bdfbaeae713c1833b3cd450857933b6e6f6ba0d0016d2cb55e16c3f0ec2fe6d)
BibTeX
@misc{dc7db7fe592799,
  title = {Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent — arXiv:2609.03221},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03221},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7db7fe592799; sha256 0bdfbaeae713c1833b3cd450857933b6e6f6ba0d0016d2cb55e16c3f0ec2fe6d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03221. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Counterfactual fairness audits of clinical language-model agents report a flip rate: how often an action changes when only the patient's demographic descriptor changes. Part of that rate is not demographic. A stochastic agent also changes its own action when nothing changes, and a flip rate cannot be interpreted without knowing how often. We measured it. Re-running one condition ten times over sixteen synthetic vignettes at default sampling changed a clinical agent's action in 8.7 percent of replicate pairs, from 2.2 percent for intensive-care escalation to 17.9 percent for controlled-substance caution, an output given no operational criteria. Across six models from five vendors, pooled floors ranged from 2.5 to 23.7 percent; in this panel neither disclosed size, vendor, nor hosting ordered them. The floor depends on the decoding configuration: majority voting over five draws removed 39 percent of it (95 percent confidence interval (CI) 18 to 64); at temperature 0 three of four locally served models showed no disagreement, but a hosted model still did. We also show that, for a binary action, the flip rate expected under no demographic effect equals the floor and a real effect adds only its square, so a flip rate inside the floor is not evidence of fairness, and direction must be tested with a signed paired test. We give a four-step reporting procedure and release FairMedAgent, the harness, with its protocol, vignettes and analysis scripts, so any team can measure the floor for its own agent.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG); Applications (stat.AP).
Pagina del lavoro: https://arxiv.org/abs/2609.03221 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.824Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).