Metraipolis › archive › dc_7feb36d5fe78
Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets — arXiv:2609.38914
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38914
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:39:40.509Z
- SHA-256 of the archived text
9c1b4d53b082c7ead57db1f0c05a7629e2064c626992c4b60a8cd1b61a2df41a- Stable URL of this page
https://metraipolis.com/documento/dc_7feb36d5fe78
How to cite
Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets — arXiv:2609.38914 — arXiv (en), https://arxiv.org/abs/2609.38914 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7feb36d5fe78 (Metraipolis archive dc_7feb36d5fe78, sha256 9c1b4d53b082c7ead57db1f0c05a7629e2064c626992c4b60a8cd1b61a2df41a)
BibTeX
@misc{dc7feb36d5fe78,
title = {Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets — arXiv:2609.38914},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38914},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7feb36d5fe78; sha256 9c1b4d53b082c7ead57db1f0c05a7629e2064c626992c4b60a8cd1b61a2df41a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38914. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating interactive agents is expensive. Agent behavior is stochastic, so reliability must be measured over repeated trials, but failures are rare and differ widely in how much they matter. Standard benchmarks spend this budget uniformly: a read-only lookup is sampled as often as an irreversible payment action. We instead formulate evaluation as a sequential allocation problem. Given a fixed trial budget and a set of scenarios whose failure behavior is unknown, which scenarios should be run, and run again? We propose a risk-aware contextual Thompson Sampling policy that combines a pre-execution scenario context vector and a fixed impact score with the failure outcomes observed during evaluation, and we test it by offline replay over 70 $\tau$-bench airline scenarios and 824 recorded trials. Our main result is at the smallest budget: with only 50 trials ($6\%$ of the corpus), the policy recovers $86\%$ of the impact-weighted failures an oracle could find, compared to $25\%$ for uniform allocation. It discovers $3.5\times$ more impact-weighted failures (215.4 vs. 62.2) with the same number of trials, delivers $5\times$ the discovery per dollar, and cuts the budget wasted on scenarios that never fail from $34\%$ to $2.8\%$. The rest of our analysis demonstrates and qualifies this result: a budget sweep shows the advantage shrinks as the budget approaches the corpus size, and paired significance tests show that scenario context helps mainly at small budgets while posterior-based exploration helps at moderate ones. Risk-aware adaptive allocation therefore helps most exactly where evaluation budget is scarcest. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.MA). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Multiagent Systems (cs.MA). Pagina del lavoro: https://arxiv.org/abs/2609.38914 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:12:18.650Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).