Metraipolis › archive › dc_cbf0e98f50de

LLJ Cards: Best practices for the Use of LLMs as Judges — arXiv:2609.24516

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24516
License (as archived)
CC0-1.0
Archived
2026-10-09T06:50:09.982Z
SHA-256 of the archived text
3693a1c213c4649c5ae0cb455cb3857a0241db5eb380f3c701ce16de976a3986
Stable URL of this page
https://metraipolis.com/documento/dc_cbf0e98f50de

How to cite

LLJ Cards: Best practices for the Use of LLMs as Judges — arXiv:2609.24516 — arXiv (en), https://arxiv.org/abs/2609.24516 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cbf0e98f50de (Metraipolis archive dc_cbf0e98f50de, sha256 3693a1c213c4649c5ae0cb455cb3857a0241db5eb380f3c701ce16de976a3986)
BibTeX
@misc{dccbf0e98f50de,
  title = {LLJ Cards: Best practices for the Use of LLMs as Judges — arXiv:2609.24516},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24516},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cbf0e98f50de; sha256 3693a1c213c4649c5ae0cb455cb3857a0241db5eb380f3c701ce16de976a3986},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24516. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these systems have been widely adopted by researchers and practitioners across a broad range of measurement tasks, driven by their strong performance, scalability, and cost-effectiveness relative to human judgment. However, a growing body of work has shown that the use of LLJs raise concerns about their validity and reliability as evaluators. Existing efforts to address these challenges have largely focused on developing bias-mitigation techniques and refining prompting strategies. While these approaches represent an important step forward, they primarily offer technical fixes and leave a more fundamental challenge unaddressed: the lack of standardized, transparent, and reproducible evaluation practices. In this paper, we introduce LLJ Cards, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations. While LLJs offer a promising path toward scalable evaluation, their effective use requires grounding in rigorous evaluation principles to ensure validity, reliability, and reproducibility. LLJ Cards addresses this need by providing a structured framework for applying these principles in the design and reporting of automated evaluations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.24516 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:45.053Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).