Metraipolis › archive › dc_a861ced76e2e
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning — arXiv:2610.01207
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01207
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T12:04:08.587Z
- SHA-256 of the archived text
cf09a7f50a5365db61af0dbd1b41370f2bfcf194d50f6af7ddb065c95d610386- Stable URL of this page
https://metraipolis.com/documento/dc_a861ced76e2e
How to cite
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning — arXiv:2610.01207 — arXiv (en), https://arxiv.org/abs/2610.01207 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a861ced76e2e (Metraipolis archive dc_a861ced76e2e, sha256 cf09a7f50a5365db61af0dbd1b41370f2bfcf194d50f6af7ddb065c95d610386)
BibTeX
@misc{dca861ced76e2e,
title = {Dependency-Aware Reward Shaping for Agentic Reinforcement Learning — arXiv:2610.01207},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01207},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a861ced76e2e; sha256 cf09a7f50a5365db61af0dbd1b41370f2bfcf194d50f6af7ddb065c95d610386},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01207. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When training large language models with reinforcement learning, terminal rewards provide little guidance about which steps matter. Common methods for assigning step credit overlook that work built on uncorrected mistakes is wasted while independent work remains valid. With only a final success/failure reward, every step in a failed episode has zero total future reward, even when it made progress. We propose Dependency-Aware Reward Shaping (DARS), which represents task progress as predicates linked by prerequisite relations and assigns step-level credit over the dependency graph. An annotator marks which predicates each step verifies, invalidates, or repairs. Verified predicates are discounted according to graph distance from the nearest broken prerequisite, while independent predicates are unaffected. Repairs update these weights based on any errors that remain; invalidated predicates need re-verification to regain credit. A fixed potential converts these annotations into signed per-step rewards. A common reward and annotation interface allows DARS to integrate with a range of reasoning and agentic training methods, such as GiGPO and ARPO/AEPO, without changing their rollout strategies or optimizers. Across five task families and models from 1.5B to 8B, DARS improves success by up to 10 points over GiGPO trained with the same budget and harness (ALFWorld), raises the WebShop task score and Search-R1 QA accuracy, complements AEPO's entropy-based training on AIME24/25 with a Python interpreter, and exceeds OmniOPD in controlled tool-free reasoning comparisons at 1.7B and 4B. Ablations show that step-level credit, dependency attenuation, and graph topology each contribute. On ALFWorld, a distilled 8B annotator matches the API annotator, enabling DARS to run efficiently without a frontier judge. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.01207 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T10:21:27.922Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).