Metraipolis › archive › dc_c8bf72b327a5

ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL — arXiv:2609.21378

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21378
License (as archived)
CC0-1.0
Archived
2026-10-10T01:27:24.258Z
SHA-256 of the archived text
9e6936a873ef7e22f5785dce81538c83dcf7b193b415593c76fba7c14f38d881
Stable URL of this page
https://metraipolis.com/documento/dc_c8bf72b327a5

How to cite

ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL — arXiv:2609.21378 — arXiv (en), https://arxiv.org/abs/2609.21378 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c8bf72b327a5 (Metraipolis archive dc_c8bf72b327a5, sha256 9e6936a873ef7e22f5785dce81538c83dcf7b193b415593c76fba7c14f38d881)
BibTeX
@misc{dcc8bf72b327a5,
  title = {ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL — arXiv:2609.21378},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21378},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c8bf72b327a5; sha256 9e6936a873ef7e22f5785dce81538c83dcf7b193b415593c76fba7c14f38d881},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21378. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning has substantially improved large language model (LLM) agents in verifiable domains, but remains difficult to apply to open-ended agent tasks, where solutions are diverse and reliable scalar rewards are hard to obtain. Recent pairwise evaluation methods alleviate reward discrimination collapse by replacing pointwise scoring with relative preferences. However, they still compress rich comparative feedback into a single trajectory-level reward, obscuring decisive intermediate steps and preventing successful behaviors from being consolidated into reusable skills. We propose ArenaFlow, a hierarchical credit propagation framework for open-ended agent reinforcement learning. ArenaFlow leverages tournament-based relative ranking to derive trajectory-level reward signals. Each comparison is further equipped with structured reflective evaluation, which reveals three types of supervision: pivotal success steps, reusable strategy skills, and usage attribution of retrieved skills. At the step level, ArenaFlow propagates trajectory-level advantages to high-confidence pivotal steps according to tournament survival depth, enabling more targeted optimization of local reasoning behaviors. At the skill level, ArenaFlow estimates skill utility from group-level usage attribution and maintains a global skill memory through utility-aware updating, pruning, and retrieval. The resulting high-utility skills further serve as policy priors for future exploration. Extensive experiments validate ArenaFlow's effectiveness on open-ended agent tasks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.21378 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:00:30.279Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).