Metraipolis › archive › dc_634d6e0dbe21
Toward Human Rights Benchmarking for LLMs: A Pilot Methodology — arXiv:2608.10268
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.10268
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T06:08:58.846Z
- SHA-256 of the archived text
56f742eac890b7a031be5468eccc48d59080f492d7ef8b030bd1911035dd6d0f- Stable URL of this page
https://metraipolis.com/documento/dc_634d6e0dbe21
How to cite
Toward Human Rights Benchmarking for LLMs: A Pilot Methodology — arXiv:2608.10268 — arXiv (en), https://arxiv.org/abs/2608.10268 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_634d6e0dbe21 (Metraipolis archive dc_634d6e0dbe21, sha256 56f742eac890b7a031be5468eccc48d59080f492d7ef8b030bd1911035dd6d0f)
BibTeX
@misc{dc634d6e0dbe21,
title = {Toward Human Rights Benchmarking for LLMs: A Pilot Methodology — arXiv:2608.10268},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.10268},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_634d6e0dbe21; sha256 56f742eac890b7a031be5468eccc48d59080f492d7ef8b030bd1911035dd6d0f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10268. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY). Pagina del lavoro: https://arxiv.org/abs/2608.10268 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:57:28.343Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).