Metraipolis › archive › dc_c3d1995fed99

No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13607
License (as archived)
CC0-1.0
Archived
2026-10-19T02:20:17.179Z
SHA-256 of the archived text
63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8
Stable URL of this page
https://metraipolis.com/documento/dc_c3d1995fed99

How to cite

No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607 — arXiv (en), https://arxiv.org/abs/2608.13607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c3d1995fed99 (Metraipolis archive dc_c3d1995fed99, sha256 63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8)
BibTeX
@misc{dcc3d1995fed99,
  title = {No Universal Signal Predicts Sample-Level LLM Regression under Version Updates — arXiv:2608.13607},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13607},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c3d1995fed99; sha256 63a433113bccd1fd8a9635525a4df022a46628f2bbc1ddeccc786b9a5ceb90b8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one. This paper studies how to predict such regressions from signals available at inference time. We compare single-model signals (confidence, logit margin, attention entropy) against cross-version signals (output KL divergence, likelihood drift, token-level KL, representation drift) under a unified added-value test that isolates each signal's gain over a confidence baseline. Across six benchmarks in three task families (multiple-choice question answering, or MCQ; math reasoning; code generation) and six model update pairs, we find that (1) signal effectiveness is task-dependent: confidence is strongest on MCQ and simpler math, while likelihood/KL signals give the most frequent gains on harder math and code; (2) no signal is universally best across model updates either; and (3) some cross-version signals stay informative even when confidence fails, including without labels, which supports a proof-of-concept selective fallback that routes high-risk samples back to the old model. Practitioners can use these task-level patterns to choose which regression signal to trust for a given update. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.13607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.719Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).