Metraipolis › archive › dc_3a77299b43fe

Speculative Evaluation of Stochastic LLMs — arXiv:2609.28560

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.28560
License (as archived)
CC0-1.0
Archived
2026-10-08T04:26:51.092Z
SHA-256 of the archived text
94634add86001a4bdee54bb4b58844383df4315b5ff921134ad27275e4750e3a
Stable URL of this page
https://metraipolis.com/documento/dc_3a77299b43fe

How to cite

Speculative Evaluation of Stochastic LLMs — arXiv:2609.28560 — arXiv (en), https://arxiv.org/abs/2609.28560 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3a77299b43fe (Metraipolis archive dc_3a77299b43fe, sha256 94634add86001a4bdee54bb4b58844383df4315b5ff921134ad27275e4750e3a)
BibTeX
@misc{dc3a77299b43fe,
  title = {Speculative Evaluation of Stochastic LLMs — arXiv:2609.28560},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.28560},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3a77299b43fe; sha256 94634add86001a4bdee54bb4b58844383df4315b5ff921134ad27275e4750e3a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28560. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetition ignores sharp differences in task-level rollout variance. We ask how to minimize the variance of a fixed-benchmark mean under an exact rollout budget. We develop Speculative Evaluation with a Hierarchical Bayesian Neyman (HBN) policy with pilot size and stage weight jointly chosen ex ante. It runs a short uniform pilot, pools per-task success counts with a hierarchical Bayesian model, and uses posterior expectations of task-level sampling variances for exact positive-integer Neyman allocation. To mitigate the pilot synchronization barrier, HBN-async speculatively executes continuations from partial pilot feedback and retains those selected by the final allocation. Across six checkpoints and 18 benchmark groups, we evaluate 107 nondegenerate benchmark-checkpoint profiles. For rollout budgets of 8-64 per task, Speculative Evaluation reduces variance relative to Uniform by 12.8%-33.6% on average across profiles, outperforming hindsight-tuned empirical and independent Bayesian baselines. Real-generation experiments that account for the pilot synchronization barrier show that HBN-async mitigates its overhead, helping translate statistical efficiency into practical evaluation benefits.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.AI).
Categorie arXiv: Machine Learning (stat.ML) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.28560 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T03:50:57.919Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).