Metraipolis › archive › dc_04e31bf66dbf

Evaluating Model Capabilities After Benchmark Saturation — arXiv:2610.20667

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20667
License (as archived)
CC0-1.0
Archived
2026-10-16T08:55:13.430Z
SHA-256 of the archived text
f29c6fce5f1cec8691200d2df1d0397e38e7f30cd9f45791d0aef07f41672151
Stable URL of this page
https://metraipolis.com/documento/dc_04e31bf66dbf

How to cite

Evaluating Model Capabilities After Benchmark Saturation — arXiv:2610.20667 — arXiv (en), https://arxiv.org/abs/2610.20667 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_04e31bf66dbf (Metraipolis archive dc_04e31bf66dbf, sha256 f29c6fce5f1cec8691200d2df1d0397e38e7f30cd9f45791d0aef07f41672151)
BibTeX
@misc{dc04e31bf66dbf,
  title = {Evaluating Model Capabilities After Benchmark Saturation — arXiv:2610.20667},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20667},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_04e31bf66dbf; sha256 f29c6fce5f1cec8691200d2df1d0397e38e7f30cd9f45791d0aef07f41672151},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20667. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated. We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items. We evaluate representative contemporary approaches for extracting information from model transcripts: pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. We find that saturated transcripts retain capability information beyond benchmark accuracy. Amongst our evaluated protocols, pairwise judging performs best, reducing prediction error by 5.3 times compared to a baseline using only saturated benchmark scores. This capability-relevant information is distributed across transcript sections, but concentrated in specific channels of information such as reasoning and tool-calls. However, the predictive power of protocols does not always generalise across distribution shifts -- for example by underestimating the capabilities of the first reasoning models introduced in early 2025. Saturated benchmarks thus retain capability-predicting signal in their task transcripts, but extending the lifetime of existing benchmarks requires methods designed to extract these signals robustly.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.20667 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.637Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).