Metraipolis › archive › dc_0f53d6df4bcf

SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning — arXiv:2611.02414

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02414
License (as archived)
CC0-1.0
Archived
2026-11-03T10:32:41.167Z
SHA-256 of the archived text
5cce0bf11d473d8c345740b597a1bd3fc37920f2a5edb67667dea93319fd987b
Stable URL of this page
https://metraipolis.com/documento/dc_0f53d6df4bcf

How to cite

SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning — arXiv:2611.02414 — arXiv (en), https://arxiv.org/abs/2611.02414 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0f53d6df4bcf (Metraipolis archive dc_0f53d6df4bcf, sha256 5cce0bf11d473d8c345740b597a1bd3fc37920f2a5edb67667dea93319fd987b)
BibTeX
@misc{dc0f53d6df4bcf,
  title = {SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning — arXiv:2611.02414},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02414},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0f53d6df4bcf; sha256 5cce0bf11d473d8c345740b597a1bd3fc37920f2a5edb67667dea93319fd987b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02414. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

OPSD provides dense supervision for agentic reinforcement learning through privileged information feedback. However, existing privileged information often lacks explicit task-progression planning, while teacher-student log-probability gaps reflect shifts in model preference rather than true task progress. To address these limitations, we propose Subgoal-Guided Graph-Regularized OPSD (SGR-OPSD), a framework that combines subgoal supervision with interaction-derived evidence for turn-level credit assignment. Specifically, SGR-OPSD uses LLM-generated subgoal plans to provide dense action-level guidance, converting them into advantage weights via a constrained, KL-regularized objective. To ground credit in observed outcomes, we construct a Subgoal Progress Graph that estimates empirical returns from sampled transition frequencies. The graph compares empirical action returns against the state average to derive preferences that calibrate the teacher's credit weights. Evaluated on ALFWorld, WebShop, and Search-QA across two model scales (Qwen2.5-3B and 7B), SGR-OPSD consistently outperforms GRPO and strong self-distillation baselines, improving success rates over GRPO by up to 7.4 percentage points and reaching 85.8% on ALFWorld with Qwen2.5-7B. Ablation studies confirm that subgoal guidance and graph-based weight calibration are key to these performance gains.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02414 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:29:48.556Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).