Metraipolis › archive › dc_7d2cd02dca27
Cost-Aware Abstention for LLM-Based IT Ticket Classification — arXiv:2610.31657
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31657
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-23T15:16:49.979Z
- SHA-256 of the archived text
04c64e732660eae3faee9f9527b15b95f1023ae8582183d3363248d30d11e532- Stable URL of this page
https://metraipolis.com/documento/dc_7d2cd02dca27
How to cite
Cost-Aware Abstention for LLM-Based IT Ticket Classification — arXiv:2610.31657 — arXiv (en), https://arxiv.org/abs/2610.31657 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d2cd02dca27 (Metraipolis archive dc_7d2cd02dca27, sha256 04c64e732660eae3faee9f9527b15b95f1023ae8582183d3363248d30d11e532)
BibTeX
@misc{dc7d2cd02dca27,
title = {Cost-Aware Abstention for LLM-Based IT Ticket Classification — arXiv:2610.31657},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31657},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d2cd02dca27; sha256 04c64e732660eae3faee9f9527b15b95f1023ae8582183d3363248d30d11e532},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31657. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.31657 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-23T15:09:08.647Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).