Metraipolis › archive › dc_e1271cd00bf9

PACT: From Credit Assignment to Critic Alignment — arXiv:2609.26355

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.26355
License (as archived)
CC0-1.0
Archived
2026-10-08T13:47:08.946Z
SHA-256 of the archived text
419a8508d0ddf074f31345a7fc8c0c3becf16c4e0cd696a99a21915414999a97
Stable URL of this page
https://metraipolis.com/documento/dc_e1271cd00bf9

How to cite

PACT: From Credit Assignment to Critic Alignment — arXiv:2609.26355 — arXiv (en), https://arxiv.org/abs/2609.26355 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e1271cd00bf9 (Metraipolis archive dc_e1271cd00bf9, sha256 419a8508d0ddf074f31345a7fc8c0c3becf16c4e0cd696a99a21915414999a97)
BibTeX
@misc{dce1271cd00bf9,
  title = {PACT: From Credit Assignment to Critic Alignment — arXiv:2609.26355},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.26355},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e1271cd00bf9; sha256 419a8508d0ddf074f31345a7fc8c0c3becf16c4e0cd696a99a21915414999a97},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26355. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.26355 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.308Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).