Metraipolis › archive › dc_082d10bee64b

Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00474
License (as archived)
CC0-1.0
Archived
2026-11-03T18:11:59.929Z
SHA-256 of the archived text
069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440
Stable URL of this page
https://metraipolis.com/documento/dc_082d10bee64b

How to cite

Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474 — arXiv (en), https://arxiv.org/abs/2611.00474 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_082d10bee64b (Metraipolis archive dc_082d10bee64b, sha256 069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440)
BibTeX
@misc{dc082d10bee64b,
  title = {Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation — arXiv:2611.00474},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00474},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_082d10bee64b; sha256 069f467f7dbae3e49e96f07edfcd26ff991c1f695904ccafbc92ec10b445e440},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00474. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating a stochastic AI system often requires many repeated samples per instance, especially for metrics defined over long horizons. Recent work has shown that modeling instance-level success probabilities with a Beta distribution recovers Pass@$k$ from far fewer samples than direct estimation. We show that this framework supports a broader range of evaluation targets. We fit a mixture of Beta distributions by marginal maximum likelihood, with the number of components chosen by BIC. From that one fit, we compute Pass$^k$ and AUC@$k$ at any horizon in closed form. We also derive delta-method confidence intervals for these metrics and for Pass@$k$, which require no additional model calls. Treating configurations rather than tasks as instances extends the same model to sensitivity analysis. For repeated success metrics, we find that mixture priors yield more accurate estimators than naive estimation across competition mathematics and graduate-level science benchmarks when extrapolating beyond the observed number of trials, and the resulting intervals cover the full-data reference. When prompts are treated as instances, it estimates the distribution of prompt sensitivity more accurately than a comparable item-response baseline without external covariates. Direct estimators remain preferable once trials reach the target horizon.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.00474 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.287Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).