Metraipolis › archive › dc_445ad6f2a2ac
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning — arXiv:2610.23181
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.23181
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:43:08.429Z
- SHA-256 of the archived text
be805ca957085f0bc9136c0e214b7ddf0295d8994d3b505523cb6ac62199ae34- Stable URL of this page
https://metraipolis.com/documento/dc_445ad6f2a2ac
How to cite
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning — arXiv:2610.23181 — arXiv (en), https://arxiv.org/abs/2610.23181 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_445ad6f2a2ac (Metraipolis archive dc_445ad6f2a2ac, sha256 be805ca957085f0bc9136c0e214b7ddf0295d8994d3b505523cb6ac62199ae34)
BibTeX
@misc{dc445ad6f2a2ac,
title = {Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning — arXiv:2610.23181},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.23181},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_445ad6f2a2ac; sha256 be805ca957085f0bc9136c0e214b7ddf0295d8994d3b505523cb6ac62199ae34},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23181. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.23181 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:39:10.143Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).