Metraipolis › archive › dc_cffee8b5f1de

Contextual Credit Policy Optimization — arXiv:2610.19103

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19103
License (as archived)
CC0-1.0
Archived
2026-10-15T14:14:59.963Z
SHA-256 of the archived text
0c9b8c03f807fee6e11332e2174e7dfeb25d0eb171e48b6ac4a11734f1f73227
Stable URL of this page
https://metraipolis.com/documento/dc_cffee8b5f1de

How to cite

Contextual Credit Policy Optimization — arXiv:2610.19103 — arXiv (en), https://arxiv.org/abs/2610.19103 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cffee8b5f1de (Metraipolis archive dc_cffee8b5f1de, sha256 0c9b8c03f807fee6e11332e2174e7dfeb25d0eb171e48b6ac4a11734f1f73227)
BibTeX
@misc{dccffee8b5f1de,
  title = {Contextual Credit Policy Optimization — arXiv:2610.19103},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19103},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cffee8b5f1de; sha256 0c9b8c03f807fee6e11332e2174e7dfeb25d0eb171e48b6ac4a11734f1f73227},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19103. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.19103 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.524Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).