Metraipolis › archive › dc_ad76e067ff08

Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena — arXiv:2610.19095

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19095
License (as archived)
CC0-1.0
Archived
2026-10-15T14:23:17.061Z
SHA-256 of the archived text
96cd748d3707605acaac8c6e1bec15edc9799bc8b4c54f5129cf499270970876
Stable URL of this page
https://metraipolis.com/documento/dc_ad76e067ff08

How to cite

Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena — arXiv:2610.19095 — arXiv (en), https://arxiv.org/abs/2610.19095 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ad76e067ff08 (Metraipolis archive dc_ad76e067ff08, sha256 96cd748d3707605acaac8c6e1bec15edc9799bc8b4c54f5129cf499270970876)
BibTeX
@misc{dcad76e067ff08,
  title = {Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena — arXiv:2610.19095},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19095},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ad76e067ff08; sha256 96cd748d3707605acaac8c6e1bec15edc9799bc8b4c54f5129cf499270970876},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19095. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.19095 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.825Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).