Metraipolis › archive › dc_7a950c4bfa11

Dynamically Allocating Evaluation Effort for Model Ranking — arXiv:2608.03437

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03437
License (as archived)
CC0-1.0
Archived
2026-10-31T12:31:02.573Z
SHA-256 of the archived text
debc1479d0f3d6059bfc89f4ad02c9918daa25acb966099d4c539bfafbeb65ba
Stable URL of this page
https://metraipolis.com/documento/dc_7a950c4bfa11

How to cite

Dynamically Allocating Evaluation Effort for Model Ranking — arXiv:2608.03437 — arXiv (en), https://arxiv.org/abs/2608.03437 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a950c4bfa11 (Metraipolis archive dc_7a950c4bfa11, sha256 debc1479d0f3d6059bfc89f4ad02c9918daa25acb966099d4c539bfafbeb65ba)
BibTeX
@misc{dc7a950c4bfa11,
  title = {Dynamically Allocating Evaluation Effort for Model Ranking — arXiv:2608.03437},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03437},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a950c4bfa11; sha256 debc1479d0f3d6059bfc89f4ad02c9918daa25acb966099d4c539bfafbeb65ba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03437. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.03437 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:17:37.757Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).