Metraipolis › archive › dc_5a0ee78ead0d
When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift — arXiv:2609.32267
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32267
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:36:23.206Z
- SHA-256 of the archived text
957d30c98a007ad8b5022a55c3de365d12c6636c4da75576f4b43fc93a84a5b7- Stable URL of this page
https://metraipolis.com/documento/dc_5a0ee78ead0d
How to cite
When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift — arXiv:2609.32267 — arXiv (en), https://arxiv.org/abs/2609.32267 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5a0ee78ead0d (Metraipolis archive dc_5a0ee78ead0d, sha256 957d30c98a007ad8b5022a55c3de365d12c6636c4da75576f4b43fc93a84a5b7)
BibTeX
@misc{dc5a0ee78ead0d,
title = {When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift — arXiv:2609.32267},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32267},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5a0ee78ead0d; sha256 957d30c98a007ad8b5022a55c3de365d12c6636c4da75576f4b43fc93a84a5b7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32267. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Releasing a model update requires certifying that its current-population risk stays below a threshold. Trusted labels are expensive, while a cheap evaluator, such as an LLM judge, scores every example. Reusing evaluator errors from earlier audits is tempting, but when may such evidence replace current labels? It depends on the status of history. If the errors can change invisibly, no label-free test detects the change, and every valid, useful certifier must keep buying labels at a rate we characterize; if a bound on the change is assumed, label-free certification is valid at an explicit error cost. For the middle ground, where history is informative but untrusted, we propose \emph{portfolio vigilance}, a sequential certifier mixing a betting expert guided by history with one that learns only from current labels; history affects only how it bets, so validity holds for any history. The contribution is not prior-informed betting or expert mixtures, but separating history that may enter validity from history that may only guide label collection. In a canonical model, accurate history shortens decisions but never raises the evidence growth rate; stale history can destroy it. On held-out CIFAR-10N and DICES-990 data, portfolio vigilance needs 0.465 (95\% CI $[0.327,0.575]$) and 0.740 ($[0.618,0.877]$) times the labels of a matched prediction-powered monitor, with no observed false certification, and fewer labels on all six external blocks. Under corrupted advice it stays within 8.0\% of its better component, while trusting history alone costs up to 1.66 times as much. In post-confirmatory repeated-judge experiments on DICES-990 and ToxicChat, changing a fixed LLM judge's rubric moves its scores beyond run-to-run variation; the portfolio then needs 0.790 ($[0.667,0.909]$) and 0.631 ($[0.520,0.770]$) times the labels of the matched monitor, and fewer than trusting history alone.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32267 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.626Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).