Metraipolis › archive › dc_32ba9979648b

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards — arXiv:2609.13866

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.13866
License (as archived)
CC0-1.0
Archived
2026-10-16T00:10:31.809Z
SHA-256 of the archived text
d265982154aff521ef6db43df085e389efb538a85d0eb9f1716781ac5ef32c08
Stable URL of this page
https://metraipolis.com/documento/dc_32ba9979648b

How to cite

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards — arXiv:2609.13866 — arXiv (en), https://arxiv.org/abs/2609.13866 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_32ba9979648b (Metraipolis archive dc_32ba9979648b, sha256 d265982154aff521ef6db43df085e389efb538a85d0eb9f1716781ac5ef32c08)
BibTeX
@misc{dc32ba9979648b,
  title = {The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards — arXiv:2609.13866},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13866},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_32ba9979648b; sha256 d265982154aff521ef6db43df085e389efb538a85d0eb9f1716781ac5ef32c08},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13866. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in {0.1, 0.3, 0.5}, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.13866 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T23:54:14.648Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).