Metraipolis › archive › dc_6aa3cbaeabca

Breaking Bureaucracy: Evaluating open-source LLMs for legal document review — arXiv:2610.06345

Source
arXiv
Original
https://arxiv.org/abs/2610.06345
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:31.886Z
SHA-256 of the archived text
1a9f5e0b5cf081f1213811768b062c0f8c97ed1a7916824de5b75176449d378b
Stable URL of this page
https://metraipolis.com/documento/dc_6aa3cbaeabca

How to cite

Breaking Bureaucracy: Evaluating open-source LLMs for legal document review — arXiv:2610.06345 — arXiv, https://arxiv.org/abs/2610.06345 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6aa3cbaeabca (Metraipolis archive dc_6aa3cbaeabca, sha256 1a9f5e0b5cf081f1213811768b062c0f8c97ed1a7916824de5b75176449d378b)
BibTeX
@misc{dc6aa3cbaeabca,
  title = {Breaking Bureaucracy: Evaluating open-source LLMs for legal document review — arXiv:2610.06345},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06345},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6aa3cbaeabca; sha256 1a9f5e0b5cf081f1213811768b062c0f8c97ed1a7916824de5b75176449d378b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06345. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.06345 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.367Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).