Metraipolis › archive › dc_fa653d87b3b5
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers — arXiv:2609.18204
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.18204
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T13:49:22.724Z
- SHA-256 of the archived text
e50c27983d654127c393df53adbaa22fd197fe5d8675422cce715a6ea6957f2b- Stable URL of this page
https://metraipolis.com/documento/dc_fa653d87b3b5
How to cite
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers — arXiv:2609.18204 — arXiv (en), https://arxiv.org/abs/2609.18204 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa653d87b3b5 (Metraipolis archive dc_fa653d87b3b5, sha256 e50c27983d654127c393df53adbaa22fd197fe5d8675422cce715a6ea6957f2b)
BibTeX
@misc{dcfa653d87b3b5,
title = {Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers — arXiv:2609.18204},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.18204},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa653d87b3b5; sha256 e50c27983d654127c393df53adbaa22fd197fe5d8675422cce715a6ea6957f2b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18204. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Organizations increasingly use oversight loops where one large language model (LLM) audits another's outputs alongside procedural traces of claimed steps. A common concern about such LLM-as-a-judge pipelines is that detailed traces make overseers gullible. Using signal detection theory, we audit five LLM overseers on 19 compliance tasks (4,551 analyzed judgments), varying only trace detail and evidence labeling. With disconfirming evidence always visible, error detection remains near ceiling. Instead, elaborate traces shift the decision criterion toward rejection, increasing false alarms in susceptible overseers. Without option labels, human-validated reason coding shows about 60% of false alarms cite an inability to tie evidence to its option. Labels eliminate this stated reason, yet residual rejection of correct work persists in those overseers and rises with trace detail. Procedural traces thus act as governance artifacts that shape oversight decisions. AI auditors should be evaluated by their decision criterion and false-alarm behavior, alongside accuracy. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC). Pagina del lavoro: https://arxiv.org/abs/2609.18204 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:48:24.306Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).