Metraipolis › archive › dc_1c66b60fc271

When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits — arXiv:2609.24194

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24194
License (as archived)
CC0-1.0
Archived
2026-10-08T15:08:06.362Z
SHA-256 of the archived text
39bd6cc635c6467657336f0a84b41c2f83880003d096679a4624c0775a124986
Stable URL of this page
https://metraipolis.com/documento/dc_1c66b60fc271

How to cite

When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits — arXiv:2609.24194 — arXiv (en), https://arxiv.org/abs/2609.24194 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1c66b60fc271 (Metraipolis archive dc_1c66b60fc271, sha256 39bd6cc635c6467657336f0a84b41c2f83880003d096679a4624c0775a124986)
BibTeX
@misc{dc1c66b60fc271,
  title = {When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits — arXiv:2609.24194},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24194},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1c66b60fc271; sha256 39bd6cc635c6467657336f0a84b41c2f83880003d096679a4624c0775a124986},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24194. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluation scores used around LLM systems -- including reward models, rerankers, and LLM judges -- can track surface form instead of the quality they claim to measure. When presented with a terse correct solution and a commented buggy solution for the same MBPP problem, a public preference reward model selects the correct one no better than a coin flip (0.507). Subtracting the predictable surface component from such scores is increasingly common, but removal alone does not yield a more valid measurement: the removed component may carry construct-relevant signal, and residualization cannot tell which is which. Under designed interventions -- unit-test labels with comment-only edits -- residualization attenuates the reward model's format effects by about 0.12 on both correct and buggy code, while the correct-versus-buggy margins move by less than 0.01. In observational NLI and QA settings, we freeze a held-out replication before scoring and re-evaluate it using labels from disjoint annotators; this supports only a narrower conclusion: better agreement with the construct labels on a pre-declared slice where a surface-only predictor errs, not a repaired score. Full-population agreement falls in every observational setting with a reported positive slice gain, and within-question ranking falls in every such QA setting. When construct and surface features are entangled, residualization can decorrelate a score while degrading construct alignment, and, in a controlled model, configurations just as damaging to construct alignment pass every pre-adjustment check, so no committed gate is a guarantee. We assemble these distinctions into a reporting protocol whose outcomes, refusal included, state what an adjusted score may be claimed to show: an audit-time diagnostic reported beside the construct-alignment cost it incurs, never a replacement for the raw score.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.LG).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.24194 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T14:37:29.506Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).