Metraipolis › archive › dc_a435ebf9fb2f

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges — arXiv:2610.19992

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19992
License (as archived)
CC0-1.0
Archived
2026-10-15T20:34:56.101Z
SHA-256 of the archived text
52517d68d9f2f6b5a9e558291cac72464174fe923e8752cfd5fed18ce9d271ff
Stable URL of this page
https://metraipolis.com/documento/dc_a435ebf9fb2f

How to cite

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges — arXiv:2610.19992 — arXiv (en), https://arxiv.org/abs/2610.19992 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a435ebf9fb2f (Metraipolis archive dc_a435ebf9fb2f, sha256 52517d68d9f2f6b5a9e558291cac72464174fe923e8752cfd5fed18ce9d271ff)
BibTeX
@misc{dca435ebf9fb2f,
  title = {SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges — arXiv:2610.19992},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19992},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a435ebf9fb2f; sha256 52517d68d9f2f6b5a9e558291cac72464174fe923e8752cfd5fed18ce9d271ff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19992. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.19992 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:36:18.236Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).