Metraipolis › archive › dc_c51bd2fc6ba3
TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents — arXiv:2610.41697
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41697
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T05:31:30.639Z
- SHA-256 of the archived text
e99a53a13d9a8ce48160fe4933b5fb26f3f011319fce0dc47756d1a71d7627e9- Stable URL of this page
https://metraipolis.com/documento/dc_c51bd2fc6ba3
How to cite
TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents — arXiv:2610.41697 — arXiv (en), https://arxiv.org/abs/2610.41697 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c51bd2fc6ba3 (Metraipolis archive dc_c51bd2fc6ba3, sha256 e99a53a13d9a8ce48160fe4933b5fb26f3f011319fce0dc47756d1a71d7627e9)
BibTeX
@misc{dcc51bd2fc6ba3,
title = {TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents — arXiv:2610.41697},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41697},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c51bd2fc6ba3; sha256 e99a53a13d9a8ce48160fe4933b5fb26f3f011319fce0dc47756d1a71d7627e9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41697. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Training interactive language agents from terminal outcomes poses three coupled problems: which task and state continuations to sample, how to assign delayed feedback to decisions, and how to aggregate token losses for multi-token actions. We present TSBPO, a tree-structured sampling-to-update paradigm that addresses these problems within one training loop. A likelihood-fitted Root/Branch allocator uses retained task and state evidence to target continuations likely to produce contrasting terminal outcomes under fixed rollout budgets. On the resulting replayed physical trees, trajectory advantages compare immediate sibling segments, while observation-group step advantages compare outcomes at matched task/observation states. The actor then sums clipped token surrogates over each complete response and normalizes across active turns, avoiding response-length averaging in the policy-gradient term. With a Qwen2.5-1.5B actor, fixed checkpoints obtain 97.79% seen and 96.64% unseen stochastic success on ALFWorld and an 81.07% per-episode success rate over six stochastic episodes for each of the 500 WebShop test goals. A WebShop step-term ablation yields 75.93% at the same update-100 evaluation; an ALFWorld aggregation comparison shows higher observed command validity with complete-turn weighting. These results support the evaluated training configuration on two interactive benchmarks. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.41697 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T05:18:52.136Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).