Metraipolis › archive › dc_50ec9ddafc31

Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks — arXiv:2611.07042

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07042
License (as archived)
CC0-1.0
Archived
2026-11-06T14:43:00.504Z
SHA-256 of the archived text
53ead58226c1d03a08348db8656cbea0ba3c048774c20332332fcd234cfb7c35
Stable URL of this page
https://metraipolis.com/documento/dc_50ec9ddafc31

How to cite

Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks — arXiv:2611.07042 — arXiv (en), https://arxiv.org/abs/2611.07042 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_50ec9ddafc31 (Metraipolis archive dc_50ec9ddafc31, sha256 53ead58226c1d03a08348db8656cbea0ba3c048774c20332332fcd234cfb7c35)
BibTeX
@misc{dc50ec9ddafc31,
  title = {Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks — arXiv:2611.07042},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07042},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_50ec9ddafc31; sha256 53ead58226c1d03a08348db8656cbea0ba3c048774c20332332fcd234cfb7c35},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07042. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Policy-based decision benchmarks evaluate whether systems apply written rules to financial and operational cases. Yet a rule-following response can be marked wrong when the benchmark misstates the required decision. We audit nine public policy-based decision benchmarks through source tracing, program execution, and controlled checks. We find unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions. To measure the scoring impact, we rescore unchanged model responses. On the 96 LegalBench SARA numeric cases, correcting the scorer raises acceptance by more than 80 percentage points for each of three models, without changing references or responses. In RuleArena Tax, restoring a required zero floor changes 27 of 300 references and narrows an observed model gap. A controlled federal-exemption task with explicit ownership and eligibility assumptions also shows that errors in supplied intermediate answers can affect final decisions. We present BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions for reevaluation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.07042 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:57.580Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).