Metraipolis › archive › dc_3e5aeade14bd

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning — arXiv:2608.07371

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.07371
License (as archived)
CC0-1.0
Archived
2026-10-19T09:47:13.099Z
SHA-256 of the archived text
e1ecf102eaa80a683dda95cb95ea0e7611f3f91458f1a407e47dc9b10a69f0f8
Stable URL of this page
https://metraipolis.com/documento/dc_3e5aeade14bd

How to cite

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning — arXiv:2608.07371 — arXiv (en), https://arxiv.org/abs/2608.07371 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e5aeade14bd (Metraipolis archive dc_3e5aeade14bd, sha256 e1ecf102eaa80a683dda95cb95ea0e7611f3f91458f1a407e47dc9b10a69f0f8)
BibTeX
@misc{dc3e5aeade14bd,
  title = {Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning — arXiv:2608.07371},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.07371},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e5aeade14bd; sha256 e1ecf102eaa80a683dda95cb95ea0e7611f3f91458f1a407e47dc9b10a69f0f8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07371. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.07371 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.274Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).