Metraipolis › archive › dc_0f08bcd820d1
Semantic Self-Distillation for Language Model Uncertainty — arXiv:2602.04577
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2602.04577
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T10:57:45.553Z
- SHA-256 of the archived text
4aa91c013ce520e7d177b147deec26bb413d18bdbf62a7f4ea3c784f72c42f31- Stable URL of this page
https://metraipolis.com/documento/dc_0f08bcd820d1
How to cite
Semantic Self-Distillation for Language Model Uncertainty — arXiv:2602.04577 — arXiv (en), https://arxiv.org/abs/2602.04577 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0f08bcd820d1 (Metraipolis archive dc_0f08bcd820d1, sha256 4aa91c013ce520e7d177b147deec26bb413d18bdbf62a7f4ea3c784f72c42f31)
BibTeX
@misc{dc0f08bcd820d1,
title = {Semantic Self-Distillation for Language Model Uncertainty — arXiv:2602.04577},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2602.04577},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0f08bcd820d1; sha256 4aa91c013ce520e7d177b147deec26bb413d18bdbf62a7f4ea3c784f72c42f31},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2602.04577. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models present challenges for principled uncertainty quantification, in part due to their complexity and the diversity of their outputs. Semantic dispersion, or the variance in the meaning of sampled answers, has been proposed as a useful proxy for model uncertainty, but the associated computational cost prohibits its use in latency-critical applications. We show that sampled semantic distributions can be distilled into lightweight student models which estimate a prompt-conditioned density before the language model generates an answer token. The student model predicts a semantic distribution over possible answers; the entropy of this distribution provides a prompt-level uncertainty signal, and the probability density allows answer-level reliability evaluation. Across experiments on TriviaQA and MMLU, we find our student models perform competitively relative to the teacher's sampled semantic dispersion on a hallucination prediction task, whilst offering additional uncertainty primitives for out-of-domain detection and multiple-choice answer selection. We term this technique Semantic Self-Distillation (SSD), which can serve as a general framework for distilling predictive uncertainty in complex output spaces beyond language. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2602.04577 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:06:16.787Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).