Metraipolis › archive › dc_52bc24e860f9

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition? — arXiv:2609.00482

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00482
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:17.363Z
SHA-256 of the archived text
e300f730e3d056a7104dc7dd80b9a1d58dc626af83e8cc68c638f5fb57f4c98a
Stable URL of this page
https://metraipolis.com/documento/dc_52bc24e860f9

How to cite

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition? — arXiv:2609.00482 — arXiv (en), https://arxiv.org/abs/2609.00482 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_52bc24e860f9 (Metraipolis archive dc_52bc24e860f9, sha256 e300f730e3d056a7104dc7dd80b9a1d58dc626af83e8cc68c638f5fb57f4c98a)
BibTeX
@misc{dc52bc24e860f9,
  title = {Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition? — arXiv:2609.00482},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00482},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_52bc24e860f9; sha256 e300f730e3d056a7104dc7dd80b9a1d58dc626af83e8cc68c638f5fb57f4c98a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00482. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Small leaderboard gaps are often interpreted as evidence that one language model is better than another, but their sign may depend on which benchmark items are included. We test this using item-level responses from five benchmarks and a family-label-free spectral approximation to multidimensional item-response theory (MIRT). In owner-disjoint folds, one owner half identifies items with low residual differential item functioning across model families (low-DIF). These items are used to score models in the other half with frozen weights that preserve the benchmark's composition across metadata-defined item groups and easiness strata. Equally short matched-random subtests provide a baseline for variation due to item subsampling. Full-benchmark and low-DIF rankings remain strongly correlated ($\tau_b=.900$--$.948$). Yet in four of five benchmarks, 30.9--47.1\% of cross-family pairs initially within one percentage point reverse order, exceeding their matched-random medians by 16.9--28.6 percentage points (all $p=.001$). The fifth benchmark shows no reliable excess ($-0.9$ points, $p=.689$). The pattern survives all pre-specified population perturbations, and residual item--family signatures replicate across owner halves; however, no family shows a consistent advantage across benchmarks. Thus, globally stable rankings can still leave individual near-tie orderings sensitive to benchmark composition, and sub-one-point leaderboard gaps should be accompanied by evidence that the implied ordering is composition-robust.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.00482 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.808Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).