Metraipolis › archive › dc_33fd0b063df4

Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents — arXiv:2609.33391

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33391
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:22.483Z
SHA-256 of the archived text
b20d207ecd956975da7f0c25b0390071b070b9c0dfe4dddcdb89f9eed20d29f9
Stable URL of this page
https://metraipolis.com/documento/dc_33fd0b063df4

How to cite

Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents — arXiv:2609.33391 — arXiv (en), https://arxiv.org/abs/2609.33391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33fd0b063df4 (Metraipolis archive dc_33fd0b063df4, sha256 b20d207ecd956975da7f0c25b0390071b070b9c0dfe4dddcdb89f9eed20d29f9)
BibTeX
@misc{dc33fd0b063df4,
  title = {Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents — arXiv:2609.33391},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33391},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33fd0b063df4; sha256 b20d207ecd956975da7f0c25b0390071b070b9c0dfe4dddcdb89f9eed20d29f9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) often relies on sparse outcome rewards, providing coarse supervision for long-horizon agents. On-policy self-distillation (OPSD) complements this signal with dense privileged feedback. However, we identify \emph{Decision--Timestamp Mismatch}: privileged guidance may be misaligned with the student's functional decision because the corresponding decision can occur at a different timestep, while the student's decision itself may span multiple timesteps rather than being tied to a single timestamp. Thus, timestamp-local supervision can misalign both the context and the temporal scope of credit. To address this mismatch, we introduce \textsc{AlignOPSD}, following the principle of aligning supervision before assigning credit. Decision-Aligned Supervision Rectification re-scores the same student-sampled response in functionally matched contexts across sibling rollouts to calibrate local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives variable-duration decision spans from correspondence changes and uses rectified evidence to allocate outcome-grounded credit across spans and their constituent turns. We evaluate \textsc{AlignOPSD} with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA against representative baselines. \textsc{AlignOPSD} outperforms both GRPO and StepOPSD across all eight backbone--aggregate-metric comparisons, improving on GRPO by 5.5--8.7 \% and ranking first in six. Additional analyzes examine the two alignment stages and hyperparameter sensitivity between tasks. Our code is avaliable at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.33391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.634Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).