Metraipolis › archive › dc_fcf870eaaf54

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning — arXiv:2611.05266

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05266
License (as archived)
CC0-1.0
Archived
2026-11-04T18:55:39.315Z
SHA-256 of the archived text
0de8eec16b93eeb07774a438c60f8fb550359380950ce616b0a820581ff01cd1
Stable URL of this page
https://metraipolis.com/documento/dc_fcf870eaaf54

How to cite

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning — arXiv:2611.05266 — arXiv (en), https://arxiv.org/abs/2611.05266 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcf870eaaf54 (Metraipolis archive dc_fcf870eaaf54, sha256 0de8eec16b93eeb07774a438c60f8fb550359380950ce616b0a820581ff01cd1)
BibTeX
@misc{dcfcf870eaaf54,
  title = {Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning — arXiv:2611.05266},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05266},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcf870eaaf54; sha256 0de8eec16b93eeb07774a438c60f8fb550359380950ce616b0a820581ff01cd1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05266. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.05266 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:35:37.492Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).