Metraipolis › archive › dc_c3d1995fed99
No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.13607
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T02:20:17.179Z
- SHA-256 of the archived text
63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8- Stable URL of this page
https://metraipolis.com/documento/dc_c3d1995fed99
How to cite
No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607 — arXiv (en), https://arxiv.org/abs/2608.13607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c3d1995fed99 (Metraipolis archive dc_c3d1995fed99, sha256 63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8)
BibTeX
@misc{dcc3d1995fed99,
title = {No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.13607},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c3d1995fed99; sha256 63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one. This paper studies how to predict such regressions from signals available at inference time. We compare single-model signals (confidence, logit margin, attention entropy) against cross-version signals (output KL divergence, likelihood drift, token-level KL, representation drift) under a unified added-value test that isolates each signal's gain over a confidence baseline. Across six benchmarks in three task families (multiple-choice question answering, or MCQ; math reasoning; code generation) and six model update pairs, we find that (1) signal effectiveness is task-dependent: confidence is strongest on MCQ and simpler math, while likelihood/KL signals give the most frequent gains on harder math and code; (2) no signal is universally best across model updates either; and (3) some cross-version signals stay informative even when confidence fails, including without labels, which supports a proof-of-concept selective fallback that routes high-risk samples back to the old model. Practitioners can use these task-level patterns to choose which regression signal to trust for a given update. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.13607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T02:02:02.719Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).