Metraipolis › archive › dc_0185adfa7b3a

Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change — arXiv:2611.04099

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04099
License (as archived)
CC0-1.0
Archived
2026-11-04T21:42:21.287Z
SHA-256 of the archived text
a5562cdb1c469c572ffdf6ef69729879280aa3fa169245ba9ef3777d8a3197dc
Stable URL of this page
https://metraipolis.com/documento/dc_0185adfa7b3a

How to cite

Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change — arXiv:2611.04099 — arXiv (en), https://arxiv.org/abs/2611.04099 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0185adfa7b3a (Metraipolis archive dc_0185adfa7b3a, sha256 a5562cdb1c469c572ffdf6ef69729879280aa3fa169245ba9ef3777d8a3197dc)
BibTeX
@misc{dc0185adfa7b3a,
  title = {Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change — arXiv:2611.04099},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04099},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0185adfa7b3a; sha256 a5562cdb1c469c572ffdf6ef69729879280aa3fa169245ba9ef3777d8a3197dc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04099. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

NLP evaluation reports routinely cite sliced performance differences, most prominently the intra-inter gap in document-level relation extraction, as capability evidence. We show that such gap readings are joint properties of models and evaluation contracts: scoring function, gold version, threshold, output format, and prompt language. With predictions frozen, a method ordering reverses under a gold-standard exchange (+0.0363, CI excluding zero; tie-level after), movement localizing where gold re-partitions: renames are gauge transformations (Delta=0; re-annotation +0.862). We formalize this as an evaluation leverage account centered on the rank-flip condition. Constructively, the audited evaluation reading (AER) is a contract-marginalized, tie-declared, seed-prescribed, gate-filtered procedure whose tie declarations cover every contract-induced flip on the declared WMT19 grids (14/14; held-out 73% vs. union-of-ties 89%), matching the best single metric contract at epsilon=0.10 equivalence. Evidence spans DocRE, HacRED, a UD probe, and released gold revisions; an alternative-reference-set replication extends it beyond our testbeds. Failed predictions retained as boundaries; a 5% violation gate flags unverifiable settings and bounds survivor-selection bias. We release EvalLever, a preregistered workflow with an amendment trail; on LLM-as-judge grids, the audited ensemble stabilizes every unstable pair (13-15 to zero; generic ensemble shrinkage: the preregistered correlation criterion failed, the companion stabilization criterion passed).

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04099 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:36:04.405Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).