Metraipolis › archive › dc_ddbf64ea9106

Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts? — arXiv:2611.10048

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.10048
License (as archived): CC0-1.0
Archived: 2026-11-10T18:40:42.075Z
SHA-256 of the archived text: 7148ea9af4e1c1114677eea8ca028e3468546cd8ac867f7b001116efc7221e22
Stable URL of this page: https://metraipolis.com/documento/dc_ddbf64ea9106

How to cite

Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts? — arXiv:2611.10048 — arXiv (en), https://arxiv.org/abs/2611.10048 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ddbf64ea9106 (Metraipolis archive dc_ddbf64ea9106, sha256 7148ea9af4e1c1114677eea8ca028e3468546cd8ac867f7b001116efc7221e22)

BibTeX

@misc{dcddbf64ea9106,
  title = {Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts? — arXiv:2611.10048},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.10048},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ddbf64ea9106; sha256 7148ea9af4e1c1114677eea8ca028e3468546cd8ac867f7b001116efc7221e22},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10048. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement ({\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement ({\alpha}) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.10048 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.476Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).