Metraipolis › archive › dc_3ce888011ada

Semifactual Credit-Augmented Policy Optimization — arXiv:2609.40360

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.40360
License (as archived): CC0-1.0
Archived: 2026-10-07T12:46:23.473Z
SHA-256 of the archived text: dcf87c4ee076bc742861e8c77d9598718d54ad30bbcc8becf46fe380177f8a19
Stable URL of this page: https://metraipolis.com/documento/dc_3ce888011ada

How to cite

Semifactual Credit-Augmented Policy Optimization — arXiv:2609.40360 — arXiv (en), https://arxiv.org/abs/2609.40360 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3ce888011ada (Metraipolis archive dc_3ce888011ada, sha256 dcf87c4ee076bc742861e8c77d9598718d54ad30bbcc8becf46fe380177f8a19)

BibTeX

@misc{dc3ce888011ada,
  title = {Semifactual Credit-Augmented Policy Optimization — arXiv:2609.40360},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.40360},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3ce888011ada; sha256 dcf87c4ee076bc742861e8c77d9598718d54ad30bbcc8becf46fe380177f8a19},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40360. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.40360 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:14.258Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).