Metraipolis › archive › dc_e53f82b81efc

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations — arXiv:2608.17719

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.17719
License (as archived)
CC0-1.0
Archived
2026-10-18T14:33:40.788Z
SHA-256 of the archived text
1f23ab71d1cf19aae31248bf4f15979446f95700a88ca790e6fa3d0fbd370a98
Stable URL of this page
https://metraipolis.com/documento/dc_e53f82b81efc

How to cite

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations — arXiv:2608.17719 — arXiv (en), https://arxiv.org/abs/2608.17719 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e53f82b81efc (Metraipolis archive dc_e53f82b81efc, sha256 1f23ab71d1cf19aae31248bf4f15979446f95700a88ca790e6fa3d0fbd370a98)
BibTeX
@misc{dce53f82b81efc,
  title = {What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations — arXiv:2608.17719},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.17719},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e53f82b81efc; sha256 1f23ab71d1cf19aae31248bf4f15979446f95700a88ca790e6fa3d0fbd370a98},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.17719. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration-benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.17719 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:27:54.456Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).