Metraipolis › archive › dc_134fbe89d068

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents — arXiv:2608.06312

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06312
License (as archived)
CC0-1.0
Archived
2026-10-25T21:27:27.284Z
SHA-256 of the archived text
41e935403586bb224a451055eb45fc539eb5dfebba4290ddfea31409bf313cd0
Stable URL of this page
https://metraipolis.com/documento/dc_134fbe89d068

How to cite

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents — arXiv:2608.06312 — arXiv (en), https://arxiv.org/abs/2608.06312 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_134fbe89d068 (Metraipolis archive dc_134fbe89d068, sha256 41e935403586bb224a451055eb45fc539eb5dfebba4290ddfea31409bf313cd0)
BibTeX
@misc{dc134fbe89d068,
  title = {Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents — arXiv:2608.06312},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06312},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_134fbe89d068; sha256 41e935403586bb224a451055eb45fc539eb5dfebba4290ddfea31409bf313cd0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06312. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency. Existing benchmarks focus on domain knowledge and question answering, largely overlooking intrinsic quality review for professional documents. Such reviews rely heavily on human experts, making them costly and difficult to scale. To bridge this gap, we introduce GB/T-Bench, the first benchmark for the structured review of national standard documents. Its GB/T Review Taxonomy is a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. A controllable counterexample generation mechanism combines deterministic rules and constrained LLM rewriting to process 488 documents into 7,306 traceable review error instances for evaluation. We also develop a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics. We further propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Experiments with 14 mainstream LLMs reveal a substantial human-LLM gap: the strongest model achieves only 0.3280 CMCS versus 0.6640 for experts. GB/T-Reviewer raises the best CMCS to 0.5094, showing the value of structured skill coordination for rule-intensive document review. This work paves the way for trustworthy AI in standardization and other high-stakes document domains.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-07 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.06312 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:24:38.229Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).