Metraipolis › archive › dc_d74d41e488ae
The Impact of Backbone Evolution on LLM-Based Relevance Assessments — arXiv:2610.09820
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09820
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T12:46:46.122Z
- SHA-256 of the archived text
c1e4ddd4b524432368eb041c85d1d3397e681d5c2b7fc0733da13cc698d528ac- Stable URL of this page
https://metraipolis.com/documento/dc_d74d41e488ae
How to cite
The Impact of Backbone Evolution on LLM-Based Relevance Assessments — arXiv:2610.09820 — arXiv (en), https://arxiv.org/abs/2610.09820 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d74d41e488ae (Metraipolis archive dc_d74d41e488ae, sha256 c1e4ddd4b524432368eb041c85d1d3397e681d5c2b7fc0733da13cc698d528ac)
BibTeX
@misc{dcd74d41e488ae,
title = {The Impact of Backbone Evolution on LLM-Based Relevance Assessments — arXiv:2610.09820},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09820},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d74d41e488ae; sha256 c1e4ddd4b524432368eb041c85d1d3397e681d5c2b7fc0733da13cc698d528ac},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09820. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLMs are evolving rapidly, with newer models offering stronger capabilities. This suggests that in LLM-based relevance judging, more capable models will achieve higher agreement with human judgements under the same prompt. We challenge this understanding by investigating the behavior of LLM-based relevance judges under backbone evolution. Keeping the prompts fixed, we evaluate a representative single-prompt (UMBRELA) and a rubric-based prompt (EXAM) across sequential model versions of commercial (Gemini, GPT) and open-weight (Qwen, Llama) models. Overall, we find no consistent evidence that newer versions lead to better relevance judges. Crucially, similar or improved aggregate performance does not imply judgment stability: correct judgements made by an earlier version of an LLM backbone are not necessarily preserved by later versions. We investigate the potential drivers of these regressions. Our findings caution against the assumption that judging prompts designed and validated for one backbone version will perform equivalently or better when the model is updated, even within the same family. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.IR, nuovi lavori del giorno). Categorie arXiv: Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2610.09820 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T12:18:00.201Z dalla pagina https://arxiv.org/list/cs.IR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).