Metraipolis › archive › dc_02503d49cb50

Holdout Best-of-N: Unbiased Evaluation and Its Cost — arXiv:2610.08719

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08719
License (as archived)
CC0-1.0
Archived
2026-10-07T11:23:30.581Z
SHA-256 of the archived text
829a9982afc7d93ed7978a87b62a48b830f747ebc0679304dd97200c2a6c9bea
Stable URL of this page
https://metraipolis.com/documento/dc_02503d49cb50

How to cite

Holdout Best-of-N: Unbiased Evaluation and Its Cost — arXiv:2610.08719 — arXiv (en), https://arxiv.org/abs/2610.08719 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_02503d49cb50 (Metraipolis archive dc_02503d49cb50, sha256 829a9982afc7d93ed7978a87b62a48b830f747ebc0679304dd97200c2a6c9bea)
BibTeX
@misc{dc02503d49cb50,
  title = {Holdout Best-of-N: Unbiased Evaluation and Its Cost — arXiv:2610.08719},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08719},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_02503d49cb50; sha256 829a9982afc7d93ed7978a87b62a48b830f747ebc0679304dd97200c2a6c9bea},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08719. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward. We study evaluation from a fixed matrix of $K$ independent scores per candidate for a policy that selects using $J$ fresh scores. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2$. At $J=K-1$, the selector deepens as $K$ grows. For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $\sigma^2/\sqrt K$, attained by Holdout; allowing bias improves the rate to $\sigma^2/K$. For two candidates, we derive the minimum-variance unbiased estimator at known variance and the sharp asymptotic unbiased minimax constant $1/(\pi\sqrt2)$, which Holdout attains without knowing the variance. The cyclic average over subsets and ties can be computed in $O(MK\log M)$ operations. At fixed selector depth, cyclic evaluation of bounded scores has $O(K^{-1})$ risk uniformly in pool size. The impossibility result concerns the fixed matrix: one additional fresh winner score permits unbiased evaluation of the all-$K$ policy.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.08719 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.465Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).