Metraipolis › archive › dc_3bbee19a96ab
How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification — arXiv:2609.26468
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.26468
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T14:36:46.157Z
- SHA-256 of the archived text
8e93520aa95eb4c6f887f8d55c2501b228956b2d7a2e84413228858f6061021c- Stable URL of this page
https://metraipolis.com/documento/dc_3bbee19a96ab
How to cite
How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification — arXiv:2609.26468 — arXiv (en), https://arxiv.org/abs/2609.26468 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3bbee19a96ab (Metraipolis archive dc_3bbee19a96ab, sha256 8e93520aa95eb4c6f887f8d55c2501b228956b2d7a2e84413228858f6061021c)
BibTeX
@misc{dc3bbee19a96ab,
title = {How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification — arXiv:2609.26468},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.26468},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3bbee19a96ab; sha256 8e93520aa95eb4c6f887f8d55c2501b228956b2d7a2e84413228858f6061021c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26468. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A key factor in deciding whether to trust an automatic prediction is its confidence score, which should be calibrated to match the actual probability of the prediction being correct. Most confidence calibration metrics target binary or multi-class tasks, while multi-label calibration remains largely underexplored. Multi-label classification tasks, such as assigning medical codes to clinical notes or determining news topics, are usually dominated by a large number of negatives, i.e., labels that do not apply. We show that existing binning schemes to compute label-wise expected calibration error either underestimate the error, simply reflect label frequency, or suffer from many bins with very few instances. To achieve trustworthy label-wise calibration errors, we propose a new binning scheme that gives equal weight to positive and negative label assignments. Our empirical study demonstrates that in contrast to existing binning schemes, our new scheme results in meaningful estimates of calibration error in hierarchical and in extreme multi-label classification. We also show that calibrating confidence scores of large language models for multi-label predictions is an open challenge. Our detailed analysis lays the foundation for further research by providing a solid evaluation metric for measuring calibration in multi-label classification. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.26468 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T13:46:15.752Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).