Metraipolis › archive › dc_eccebbb1818d

Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning — arXiv:2611.07639

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07639
License (as archived): CC0-1.0
Archived: 2026-11-06T14:42:30.030Z
SHA-256 of the archived text: 6484a166721fcbd31337629a0caa66a93c57c96dbf5ae3b306101417aa724545
Stable URL of this page: https://metraipolis.com/documento/dc_eccebbb1818d

How to cite

Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning — arXiv:2611.07639 — arXiv (en), https://arxiv.org/abs/2611.07639 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eccebbb1818d (Metraipolis archive dc_eccebbb1818d, sha256 6484a166721fcbd31337629a0caa66a93c57c96dbf5ae3b306101417aa724545)

BibTeX

@misc{dceccebbb1818d,
  title = {Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning — arXiv:2611.07639},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07639},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eccebbb1818d; sha256 6484a166721fcbd31337629a0caa66a93c57c96dbf5ae3b306101417aa724545},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07639. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning from verifiable rewards (RLVR) is usually evaluated as an alternative to supervised fine-tuning (SFT), or as a stage that follows it, with the implicit assumption that the RL stage contributes gains of its own. We measure that contribution directly, in a controlled setting where both stages can be run to completion and compared exactly. We built gpu-train-debug, a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks across 30 archetypes, graded by a programmatic verifier with partial credit, and calibrated so that a mid-tier model scores in a 0.60 to 0.75 band. A pilot produced a surprise: 1,200 gold demonstrations nearly solved the benchmark (SFT 0.785 on the hard split), while 15 steps of GRPO from the base model did not move it (0.314 vs 0.311 base). We therefore amended the study, in writing and before the full run, to ask the sharper question: what does verifier-based RL add beyond demonstrations? At the pre-registered scale (60 GRPO steps, learning rate 1e-6), the answer is: no detectable improvement in-distribution (+0.22 points, 95% CI [-0.06, +0.51]), and a modest but measurable improvement under distribution shift (+0.33 points on a held-out OOD split, CI [+0.02, +0.68]). SFT accounted for most of the available improvement, while verifier-based RL produced only modest additional gains under distribution shift. The residual errors SFT leaves are mostly systematic, identical across samples, which likely reduces the within-group learning signal available to GRPO. A full 60-step cold-start GRPO arm confirms the pilot at the full horizon. The whole study cost $8.69 of compute and is fully reproducible from archived generations, frozen configs, and tagged code.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.07639 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:42.681Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).