Metraipolis › archive › dc_4fa68437cd22

Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation — arXiv:2609.29140

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.29140
License (as archived): CC0-1.0
Archived: 2026-10-08T04:25:54.348Z
SHA-256 of the archived text: d0bb8ae48c064378f3006009d911783e806077c723ac536449630ce9283d7d04
Stable URL of this page: https://metraipolis.com/documento/dc_4fa68437cd22

How to cite

Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation — arXiv:2609.29140 — arXiv (en), https://arxiv.org/abs/2609.29140 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4fa68437cd22 (Metraipolis archive dc_4fa68437cd22, sha256 d0bb8ae48c064378f3006009d911783e806077c723ac536449630ce9283d7d04)

BibTeX

@misc{dc4fa68437cd22,
  title = {Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation — arXiv:2609.29140},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.29140},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4fa68437cd22; sha256 d0bb8ae48c064378f3006009d911783e806077c723ac536449630ce9283d7d04},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29140. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requirement on a fixed grid of $M$ tasks with $L$ binary paths per task under the hard budget $(M+t)K$, where each path costs at most $K$ responses or episodes. For fixed $L \ge 3$ and $0 < \alpha \le 1/12$, the optimal expected width on the worst pure cohort is $\Theta_{\alpha,L}([M(t+1)]^{-1/2})$ when every task is observed and $\Theta_{\alpha,L}([M(t+\sqrt{M})]^{-1/2})$ when omission is allowed. The lower bounds cover adaptive hard-budget policies, and fixed random-subset designs attain both rates through disagreement certificates. A joint mean/disagreement interval turns the task-covering law into practical finite-budget inference. In an equal-budget LiveCodeBench replay with 16 models, 880 tasks, and five outputs per task, the task-covering design reduces median point-estimation MSE by 87.0\% relative to pooled uniform sampling, while the Joint certificate produces narrower confidence intervals in 15/16 panels and reduces median interval width by 30.6\%. Finite-regime analyses identify task coverage as the effective choice at the evaluated scale and characterize how cohort size and within-task agreement determine the useful operating region. Together, the sharp laws and fixed-budget evidence make replication and task coverage explicit design variables for information-efficient repeated evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Information Theory (cs.IT).
Pagina del lavoro: https://arxiv.org/abs/2609.29140 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T03:50:57.719Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).