Metraipolis › archive › dc_a2d1907e015a
Cliff: Learning Process Rewards from the First Mistake — arXiv:2609.02817
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.02817
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T05:00:43.832Z
- SHA-256 of the archived text
391fc0d3b72fc1bae013ce501838b905ce257750c820bac382cf03e217db974c- Stable URL of this page
https://metraipolis.com/documento/dc_a2d1907e015a
How to cite
Cliff: Learning Process Rewards from the First Mistake — arXiv:2609.02817 — arXiv (en), https://arxiv.org/abs/2609.02817 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a2d1907e015a (Metraipolis archive dc_a2d1907e015a, sha256 391fc0d3b72fc1bae013ce501838b905ce257750c820bac382cf03e217db974c)
BibTeX
@misc{dca2d1907e015a,
title = {Cliff: Learning Process Rewards from the First Mistake — arXiv:2609.02817},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.02817},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a2d1907e015a; sha256 391fc0d3b72fc1bae013ce501838b905ce257750c820bac382cf03e217db974c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02817. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.02817 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T04:09:43.337Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).