Metraipolis › archive › dc_fae7244cfa24
What Fixed-Rollout pass@k Evaluations Can Identify — arXiv:2609.09245
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.09245
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T07:25:11.001Z
- SHA-256 of the archived text
ae8afbf6a61adc439813c77b277d4c01c7f592811c160a51a943a1f009f6334c- Stable URL of this page
https://metraipolis.com/documento/dc_fae7244cfa24
How to cite
What Fixed-Rollout pass@k Evaluations Can Identify — arXiv:2609.09245 — arXiv (en), https://arxiv.org/abs/2609.09245 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fae7244cfa24 (Metraipolis archive dc_fae7244cfa24, sha256 ae8afbf6a61adc439813c77b277d4c01c7f592811c160a51a943a1f009f6334c)
BibTeX
@misc{dcfae7244cfa24,
title = {What Fixed-Rollout pass@k Evaluations Can Identify — arXiv:2609.09245},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.09245},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fae7244cfa24; sha256 ae8afbf6a61adc439813c77b277d4c01c7f592811c160a51a943a1f009f6334c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09245. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/random-task conditional-Binomial model, fixed-n success counts identify only the n free moments of the latent per-task success distribution. Consequently, direct pass@k is identified for k <= n, but generic extrapolated pass@k, tail exponents, and tail constants are not identified for k > n, even with arbitrarily many exchangeable tasks at the same rollout budget. This is stronger than the observation that the usual estimator is undefined beyond n: it characterizes the information missing from the fixed-depth count-law experiment. We give exact count-law-preserving constructions with incompatible extrapolations, state the exceptional unique-extension case, and compute sharp population identified intervals through Hausdorff principal representations. On the public 10,000-rollout-per-problem release of Brown et al., counterfactual n = 16 evaluations leave failure at k = 1000 ambiguous by factors from 1.5 to over 2,600 across four MATH/GSM8K/CodeContests configurations. The calibration shows that intermediate-scale failure share alone does not determine width. Our result does not reject parametric inference-time scaling laws; it supplies the nonparametric baseline against which their assumptions can be evaluated. We give an exact, conservative one-coordinate finite-task confidence certificate and a reporting standard separating direct estimates, identified sets, and model-conditioned forecasts. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI). Categorie arXiv: Machine Learning (stat.ML) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.09245 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T07:00:16.289Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).