Metraipolis › archive › dc_8d3232394841

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations — arXiv:2608.14425

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.14425
License (as archived): CC0-1.0
Archived: 2026-10-19T02:20:50.134Z
SHA-256 of the archived text: e0bcaed59ed97e7be5de8dbe1aa174571b88eef6b2e11e6844dbd6fc763ec46c
Stable URL of this page: https://metraipolis.com/documento/dc_8d3232394841

How to cite

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations — arXiv:2608.14425 — arXiv (en), https://arxiv.org/abs/2608.14425 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d3232394841 (Metraipolis archive dc_8d3232394841, sha256 e0bcaed59ed97e7be5de8dbe1aa174571b88eef6b2e11e6844dbd6fc763ec46c)

BibTeX

@misc{dc8d3232394841,
  title = {Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations — arXiv:2608.14425},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14425},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d3232394841; sha256 e0bcaed59ed97e7be5de8dbe1aa174571b88eef6b2e11e6844dbd6fc763ec46c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14425. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.14425 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.820Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).