Metraipolis › archive › dc_976bdbf8a5f1

NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains — arXiv:2611.03576

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03576
License (as archived)
CC0-1.0
Archived
2026-11-03T11:11:29.121Z
SHA-256 of the archived text
604d50d48a6796d6d3aa249f95bfd6817fa8e81f24363027a422df8e6423df3c
Stable URL of this page
https://metraipolis.com/documento/dc_976bdbf8a5f1

How to cite

NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains — arXiv:2611.03576 — arXiv (en), https://arxiv.org/abs/2611.03576 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_976bdbf8a5f1 (Metraipolis archive dc_976bdbf8a5f1, sha256 604d50d48a6796d6d3aa249f95bfd6817fa8e81f24363027a422df8e6423df3c)
BibTeX
@misc{dc976bdbf8a5f1,
  title = {NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains — arXiv:2611.03576},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03576},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_976bdbf8a5f1; sha256 604d50d48a6796d6d3aa249f95bfd6817fa8e81f24363027a422df8e6423df3c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03576. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce NitiBench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 26 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it twenty of the 26 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.03576 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:52.368Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).