Metraipolis › archive › dc_082d10bee64b
Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00474
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:11:59.929Z
- SHA-256 of the archived text
069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440- Stable URL of this page
https://metraipolis.com/documento/dc_082d10bee64b
How to cite
Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474 — arXiv (en), https://arxiv.org/abs/2611.00474 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_082d10bee64b (Metraipolis archive dc_082d10bee64b, sha256 069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440)
BibTeX
@misc{dc082d10bee64b,
title = {Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00474},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_082d10bee64b; sha256 069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00474. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating a stochastic AI system often requires many repeated samples per instance, especially for metrics defined over long horizons. Recent work has shown that modeling instance-level success probabilities with a Beta distribution recovers Pass@$k$ from far fewer samples than direct estimation. We show that this framework supports a broader range of evaluation targets. We fit a mixture of Beta distributions by marginal maximum likelihood, with the number of components chosen by BIC. From that one fit, we compute Pass$^k$ and AUC@$k$ at any horizon in closed form. We also derive delta-method confidence intervals for these metrics and for Pass@$k$, which require no additional model calls. Treating configurations rather than tasks as instances extends the same model to sensitivity analysis. For repeated success metrics, we find that mixture priors yield more accurate estimators than naive estimation across competition mathematics and graduate-level science benchmarks when extrapolating beyond the observed number of trials, and the resulting intervals cover the full-data reference. When prompts are treated as instances, it estimates the distribution of prompt sensitivity more accurately than a comparable item-response baseline without external covariates. Direct estimators remain preferable once trials reach the target horizon. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.00474 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.287Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).