Metraipolis › archive › dc_fbbb973da975

StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation — arXiv:2610.41333

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41333
License (as archived): CC0-1.0
Archived: 2026-11-03T05:40:47.266Z
SHA-256 of the archived text: 8ab65f14b12e3b53a284e083b521b32542b38044f3ef3b117904bb699f40feb3
Stable URL of this page: https://metraipolis.com/documento/dc_fbbb973da975

How to cite

StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation — arXiv:2610.41333 — arXiv (en), https://arxiv.org/abs/2610.41333 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbbb973da975 (Metraipolis archive dc_fbbb973da975, sha256 8ab65f14b12e3b53a284e083b521b32542b38044f3ef3b117904bb699f40feb3)

BibTeX

@misc{dcfbbb973da975,
  title = {StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation — arXiv:2610.41333},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41333},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbbb973da975; sha256 8ab65f14b12e3b53a284e083b521b32542b38044f3ef3b117904bb699f40feb3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41333. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Process reward models (PRMs) score reasoning steps, but best-of-N selection also requires a rule that aggregates those scores. We study that rule as a separate evaluation object. StrictPRMBench fixes mathematical reasoning pools, scorer evidence, and the source trace of a deletion intervention, so that different reducers face the same recorded selection problem. Across seven scoring checkpoints and two benchmarks, the stored N=32 outcomes give higher deletion-induced selection instability for product than geometric mean (GM) in all 14 cells. The equal-cell difference is +0.0965; a shared-problem bootstrap, conditional on these checkpoints and frozen pools, gives a 95% interval of [0.0846,0.1086]. A pre-injection-defined sensitivity excludes replacement slots that are either reducer's clean winner and retains 4,459 of 4,622 cell-problem records. The retained difference is +0.0986 [0.0875,0.1098]. Effects are heterogeneous: the two Qwen cells have positive percentile-bootstrap intervals on retained support but two-sided exact paired p-values of 0.0625 and 0.0703. Clean-pool Qwen accuracy does not detect a product advantage, but this is not an equivalence result. We provide versioned outcome records, CPU reanalysis, and an explicit evidence-availability contract. The contribution is a diagnostic evaluation protocol and a reproducible audit of recorded selector behavior, not a new PRM, a universally optimal reducer, or evidence of deployment-wide harm. Raw-score replay and outcome-record reanalysis are distinguished: missing primary edited-score vectors limit independent evaluation of new reducers under the same stress.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.41333 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T05:26:25.682Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).