Metraipolis › archive › dc_c8aff8dc5196
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation — arXiv:2608.14329
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14329
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T02:21:31.469Z
- SHA-256 of the archived text
ac64c40b5d8654513c46c6406c875aae827d08eba5f06d7ce6d2f2bebd04ac2f- Stable URL of this page
https://metraipolis.com/documento/dc_c8aff8dc5196
How to cite
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation — arXiv:2608.14329 — arXiv (en), https://arxiv.org/abs/2608.14329 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c8aff8dc5196 (Metraipolis archive dc_c8aff8dc5196, sha256 ac64c40b5d8654513c46c6406c875aae827d08eba5f06d7ce6d2f2bebd04ac2f)
BibTeX
@misc{dcc8aff8dc5196,
title = {A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation — arXiv:2608.14329},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14329},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c8aff8dc5196; sha256 ac64c40b5d8654513c46c6406c875aae827d08eba5f06d7ce6d2f2bebd04ac2f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14329. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Principle-based regulation, with evaluative standards such as "fair, clear, and not misleading" or "deliver good outcomes", cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, adversarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regulation. We also introduce Ceca (Calibrated Exemplar-Cluster Assessment): a calibrated, auditable assessor that emits exact per-exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74 to 0.27) on keyword-stuffed Consumer Duty inputs: "compliance theatre." A second judge from a different model family agrees only at Cohen's kappa = 0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI). Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.14329 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T02:02:02.935Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).