Metraipolis › archive › dc_87d259ef9bf5
Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization — arXiv:2608.31077
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.31077
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:46:59.164Z
- SHA-256 of the archived text
ca676b5bbcd760e931619735f663942a5eb9364ca5c9b156583c1eb5bc70473a- Stable URL of this page
https://metraipolis.com/documento/dc_87d259ef9bf5
How to cite
Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization — arXiv:2608.31077 — arXiv (en), https://arxiv.org/abs/2608.31077 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_87d259ef9bf5 (Metraipolis archive dc_87d259ef9bf5, sha256 ca676b5bbcd760e931619735f663942a5eb9364ca5c9b156583c1eb5bc70473a)
BibTeX
@misc{dc87d259ef9bf5,
title = {Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization — arXiv:2608.31077},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.31077},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_87d259ef9bf5; sha256 ca676b5bbcd760e931619735f663942a5eb9364ca5c9b156583c1eb5bc70473a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.31077. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.31077 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.640Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).