Metraipolis › archive › dc_cd16b3f96a18
Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media — arXiv:2608.26138
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26138
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T15:18:11.976Z
- SHA-256 of the archived text
b27f5d81d684b787283b77d38ff2b66d5b17a78b36f95ee981b11f2dfd0fa77d- Stable URL of this page
https://metraipolis.com/documento/dc_cd16b3f96a18
How to cite
Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media — arXiv:2608.26138 — arXiv (en), https://arxiv.org/abs/2608.26138 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd16b3f96a18 (Metraipolis archive dc_cd16b3f96a18, sha256 b27f5d81d684b787283b77d38ff2b66d5b17a78b36f95ee981b11f2dfd0fa77d)
BibTeX
@misc{dccd16b3f96a18,
title = {Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media — arXiv:2608.26138},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26138},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd16b3f96a18; sha256 b27f5d81d684b787283b77d38ff2b66d5b17a78b36f95ee981b11f2dfd0fa77d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26138. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We introduce the Cross-Platform Fairness Evaluation (CPFE) framework -- a five-axis audit protocol covering discriminative performance, calibration, statistical significance, prediction equity, and attribution stability -- and apply it to four transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on a Kaggle mental health corpus (n=35,556) and evaluated on Reddit (n=6,257) and Twitter (n=2,883) test sets with emotion labels mapped to clinical proxies. All three independently evaluated models exhibit consistent and substantial cross-platform AUC degradation (30.3-35.4% on Reddit, 37.9-39.5% on Twitter) relative to within-platform performance (AUC 0.983-0.987), confirmed across five independent training seeds. Calibration failure is concurrent and severe: ECE rises from 0.056-0.060 in-domain to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter. Platform-specific temperature scaling reduces mean ECE by 88.0% without altering discriminative performance (mean |delta AUC|<0.01), confirming separable failure modes. Prediction equity analysis reveals large cross-platform disparities (raw DI < 0.17; prior-shift-adjusted DI: 0.11-0.29 on Reddit), with equalized odds differences of 0.753-0.830 for mental health proxy classes on Reddit and 0.755-0.831 for anxiety on Twitter. Attribution stability analysis shows near-complete vocabulary divergence across platforms (Jaccard J=0 in 14/16 model-class pairs at K=10). These findings support treating cross-platform validation across all five CPFE axes as a standard requirement for mental health NLP systems in heterogeneous environments. In a single-seed fine-tuning experiment, mean AUC improved by 0.216, suggesting target-platform labels provide greater benefit as training signal than as calibration signal. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.26138 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T15:15:18.032Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).