Metraipolis › archive › dc_d8e63ea1038c
JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation — arXiv:2608.02620
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.02620
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:30:50.304Z
- SHA-256 of the archived text
2c891a10e928c4f8598df7793d27e569a21649531c6915b30ece33ee5d11df40- Stable URL of this page
https://metraipolis.com/documento/dc_d8e63ea1038c
How to cite
JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation — arXiv:2608.02620 — arXiv (en), https://arxiv.org/abs/2608.02620 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d8e63ea1038c (Metraipolis archive dc_d8e63ea1038c, sha256 2c891a10e928c4f8598df7793d27e569a21649531c6915b30ece33ee5d11df40)
BibTeX
@misc{dcd8e63ea1038c,
title = {JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation — arXiv:2608.02620},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.02620},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d8e63ea1038c; sha256 2c891a10e928c4f8598df7793d27e569a21649531c6915b30ece33ee5d11df40},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02620. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices--the benchmark, the judge model, the prompt, the inference backend--affect the conclusions we draw about model quality. We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, this http URL , or OpenRouter can serve as both candidate and judge. Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.02620 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:17:37.717Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).