Metraipolis › archive › dc_fd513fc5ea71

Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification — arXiv:2608.04899

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.04899
License (as archived): CC0-1.0
Archived: 2026-10-31T11:25:19.045Z
SHA-256 of the archived text: a3f99e5ce8f50c4e231d87a16e88d0c102ff8a6b31602d58a24d6a5b7525c474
Stable URL of this page: https://metraipolis.com/documento/dc_fd513fc5ea71

How to cite

Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification — arXiv:2608.04899 — arXiv (en), https://arxiv.org/abs/2608.04899 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd513fc5ea71 (Metraipolis archive dc_fd513fc5ea71, sha256 a3f99e5ce8f50c4e231d87a16e88d0c102ff8a6b31602d58a24d6a5b7525c474)

BibTeX

@misc{dcfd513fc5ea71,
  title = {Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification — arXiv:2608.04899},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04899},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd513fc5ea71; sha256 a3f99e5ce8f50c4e231d87a16e88d0c102ff8a6b31602d58a24d6a5b7525c474},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04899. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs. For instance, Qwen3-32B verbalizes only eight unique confidence values on SST-2, with over half being exactly 95%, a pattern we observe consistently across four datasets and two LLMs. Besides limiting practical utility, we show that this sparsity critically affects evaluation: the choice of interpolation in area under the accuracy-rejection curve (AUARC) dramatically alters rankings, with consistency sampling dropping from best to worst under stepwise versus linear interpolation. We advocate for standardizing stepwise interpolation for a fairer comparison. Under such a fair evaluation, we find that weighting verbalized digits by token probabilities, a method we term verbalization logprobs, addresses sparsity and achieves the best AUARC (+2.3 points over vanilla verbalization) without incurring additional inference cost.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.04899 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:13:23.190Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).