Metraipolis › archive › dc_2bad68fe393e
Semantic Uncertainty Quantification Needs Factual Equivalence — arXiv:2609.34967
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34967
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:06.987Z
- SHA-256 of the archived text
0da8701e18e49b65395ded8d54d18fdec020b5cae05ca32463c3f113559a5e12- Stable URL of this page
https://metraipolis.com/documento/dc_2bad68fe393e
How to cite
Semantic Uncertainty Quantification Needs Factual Equivalence — arXiv:2609.34967 — arXiv (en), https://arxiv.org/abs/2609.34967 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2bad68fe393e (Metraipolis archive dc_2bad68fe393e, sha256 0da8701e18e49b65395ded8d54d18fdec020b5cae05ca32463c3f113559a5e12)
BibTeX
@misc{dc2bad68fe393e,
title = {Semantic Uncertainty Quantification Needs Factual Equivalence — arXiv:2609.34967},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34967},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2bad68fe393e; sha256 0da8701e18e49b65395ded8d54d18fdec020b5cae05ca32463c3f113559a5e12},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34967. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Semantic uncertainty quantification for large language models rests on a common template: sample several answers, measure how much they agree, and treat disagreement as uncertainty. We first formalize this template as two separate roles: an operator that compares two answers, and an aggregator that combines all pairwise comparisons into a scalar. Existing methods differ almost entirely in how they aggregate, while taking the operator off the shelf, typically an NLI model or a generic sentence encoder. We show that this reliance on off-the-shelf operators is the primary bottleneck of semantic UQ: they do not accurately measure factual equivalence of multiple answers to the same question. We resolve this with a deliberately simple recipe: a single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings. Integrating the resulting operator into existing methods improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models. The best variant reaches 0.76 mean AUROC against 0.68 for the strongest baseline, while replacing the quadratic cross-encoder comparisons of entailment-based operators with one encoder pass per answer. The uniformity of the improvement supports the view that the operator, not the aggregator, is the limiting factor. The same operator also improves single generation token-level estimators: the norm it assigns to each token measures how much that token bears on the answer, and reweighting token log-likelihoods accordingly sharpens the estimate. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.34967 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.208Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).