Metraipolis › archive › dc_8dd1bc4f1d4e

UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning — arXiv:2609.34810

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34810
License (as archived)
CC0-1.0
Archived
2026-10-07T20:58:12.496Z
SHA-256 of the archived text
6e544e0f4e75f55406473d49893f0ee7ddebdf9c00cd5d277f634970de3c1a22
Stable URL of this page
https://metraipolis.com/documento/dc_8dd1bc4f1d4e

How to cite

UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning — arXiv:2609.34810 — arXiv (en), https://arxiv.org/abs/2609.34810 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8dd1bc4f1d4e (Metraipolis archive dc_8dd1bc4f1d4e, sha256 6e544e0f4e75f55406473d49893f0ee7ddebdf9c00cd5d277f634970de3c1a22)
BibTeX
@misc{dc8dd1bc4f1d4e,
  title = {UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning — arXiv:2609.34810},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34810},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8dd1bc4f1d4e; sha256 6e544e0f4e75f55406473d49893f0ee7ddebdf9c00cd5d277f634970de3c1a22},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34810. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning has become an effective approach to training language model agents, but sparse and delayed outcome rewards provide limited guidance for credit assignment across long interaction sequences. Recent work on on-policy self-distillation (OPSD) offers complementary supervision by evaluating a policy's sampled responses under privileged training-time context. However, our diagnostics show that positive average agreement between outcome and hindsight feedback coexists with substantial local disagreement, raising the question of how to allocate influence between them at each decision. We introduce UniOPSD (Unified On-Policy Self-Distillation), which unifies these feedback sources through adaptive local credit arbitration. UniOPSD constructs comparable credit estimates from environmental returns and successful-peer hindsight at shared interaction anchors. Historical agreement determines the global mixing level, while current signal availability and relative precision adjust each source's influence at individual decisions. The episode-level outcome contribution is retained, and bounded token modulation refines the fused step credit for policy optimization. With Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, UniOPSD achieves ALFWorld success rates of $82.8\%$ and $83.6\%$, WebShop success rates of $75.0\%$ and $82.0\%$, and Search-QA aggregate accuracies of $45.3\%$ and $49.8\%$, respectively. On 3B WebShop, UniOPSD improves over SDAR by $7.0$ percentage points. Our code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34810 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.672Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).