Metraipolis › archive › dc_5fa320d69c72

Reinforcement Learning with Decomposed Subtasks — arXiv:2609.27035

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27035
License (as archived)
CC0-1.0
Archived
2026-10-08T09:11:01.743Z
SHA-256 of the archived text
1bdbb03fa847e083e0ad1713996dfc01a43ae8fbfc8557903a172ba4f3891e9b
Stable URL of this page
https://metraipolis.com/documento/dc_5fa320d69c72

How to cite

Reinforcement Learning with Decomposed Subtasks — arXiv:2609.27035 — arXiv (en), https://arxiv.org/abs/2609.27035 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5fa320d69c72 (Metraipolis archive dc_5fa320d69c72, sha256 1bdbb03fa847e083e0ad1713996dfc01a43ae8fbfc8557903a172ba4f3891e9b)
BibTeX
@misc{dc5fa320d69c72,
  title = {Reinforcement Learning with Decomposed Subtasks — arXiv:2609.27035},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27035},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5fa320d69c72; sha256 1bdbb03fa847e083e0ad1713996dfc01a43ae8fbfc8557903a172ba4f3891e9b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27035. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.27035 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.460Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).