Metraipolis › archive › dc_373a4efeef06

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing — arXiv:2610.35698

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.35698
License (as archived)
CC0-1.0
Archived
2026-10-27T08:12:29.847Z
SHA-256 of the archived text
b19535150080fbf26d3063acde658d59a4964e937ef66f93ec99ff97a300f1d9
Stable URL of this page
https://metraipolis.com/documento/dc_373a4efeef06

How to cite

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing — arXiv:2610.35698 — arXiv (en), https://arxiv.org/abs/2610.35698 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_373a4efeef06 (Metraipolis archive dc_373a4efeef06, sha256 b19535150080fbf26d3063acde658d59a4964e937ef66f93ec99ff97a300f1d9)
BibTeX
@misc{dc373a4efeef06,
  title = {From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing — arXiv:2610.35698},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.35698},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_373a4efeef06; sha256 b19535150080fbf26d3063acde658d59a4964e937ef66f93ec99ff97a300f1d9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.35698. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still be useful for efficient evaluation. At the same time, LLM-as-a-judge offers a scalable alternative to direct human auditing, but its practical benefit depends on calibration quality and inference cost. This paper develops a unified statistical framework for both evaluation routes under alternative sampling strategies. For direct auditing, confidence-stratified sampling is used to recover population accuracy from partially verified outputs without requiring calibrated confidence. Finite-sample uncertainty is characterized, and the analysis shows how highly uneven confidence strata affect audit efficiency and label allocation. For judge-assisted evaluation, the judge is modeled through sensitivity and false-positive rate, yielding an error-corrected estimator with calibration uncertainty. The two routes are then compared through human-label and total-cost efficiency boundaries. Experiments on MMLU-Pro and MMMU-Pro show that miscalibrated self-confidence can still support statistically valid stratified auditing. Under highly imbalanced confidence distributions, proportional allocation substantially reduces estimation error relative to equal-per-stratum allocation. For judge-assisted evaluation, only a small subset of target--verifier pairs satisfies the idealized human-label efficiency condition, and automated inference costs can further limit potential savings. These findings clarify the statistical and economic conditions under which confidence-guided auditing and automated verification are useful.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.35698 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.277Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).