Metraipolis › archive › dc_23765a034453

Can LLMs Reliably Evaluate Themselves? A Probabilistic VC Framework — arXiv:2611.02253

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02253
License (as archived): CC0-1.0
Archived: 2026-11-03T10:53:26.458Z
SHA-256 of the archived text: cb237632c0653d3ef0ec76be5ac48b7f1cb132f828f30615602b76b4444a70e1
Stable URL of this page: https://metraipolis.com/documento/dc_23765a034453

How to cite

Can LLMs Reliably Evaluate Themselves? A Probabilistic VC Framework — arXiv:2611.02253 — arXiv (en), https://arxiv.org/abs/2611.02253 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_23765a034453 (Metraipolis archive dc_23765a034453, sha256 cb237632c0653d3ef0ec76be5ac48b7f1cb132f828f30615602b76b4444a70e1)

BibTeX

@misc{dc23765a034453,
  title = {Can LLMs Reliably Evaluate Themselves? A Probabilistic VC Framework — arXiv:2611.02253},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02253},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_23765a034453; sha256 cb237632c0653d3ef0ec76be5ac48b7f1cb132f828f30615602b76b4444a70e1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02253. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models (LLMs) find increasing use in critical applications, evaluating their ability to assess their own outputs has become crucial. We present a theoretical and empirical framework that examines whether LLMs can differentiate between correct and incorrect solutions while maintaining properly calibrated confidence. Building on classical Vapnik-Chervonenkis (VC) theory, we adapt it to probabilistic predictors through two complexity metrics: Probabilistic VC (PVC), which measures a model's ability to confidently classify across problem types, and Calibration-aware PVC (C-PVC), which additionally demands alignment between confidence scores and actual success rates. Unlike Expected Calibration Error (ECE) and Actual Error (AE), these measures give an integrated assessment of self-evaluation expressiveness and calibration, with sample complexity bounds and generalization guarantees comparable to classical VC theory. We test eleven 7-8B models on 360 original mathematical reasoning problems, TruthfulQA, and CommonsenseQA. Each model chooses between two of its own generated solutions and reports its confidence, with ground truth determined by an ensemble of larger models. The results show a systematic inverse relationship: models with greater self-evaluation expressiveness have weaker calibration. s1.1-7B and Qwen2.5-7B-Instruct achieve the highest PVC-VUS scores, while JiuZhang3.0-7B is best calibrated, with the lowest ECE and the smallest PVC-VUS gap. The trade-off persists across architectures, training paradigms, and domains, pointing to a fundamental challenge in developing self-reflective LLMs, and our framework offers practical tools for diagnosing it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02253 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.341Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).