Metraipolis › archive › dc_37642ce43d3f
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation — arXiv:2608.04788
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.04788
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:14:38.830Z
- SHA-256 of the archived text
ac454e8a5b42220ab8946b86089db0e3b86da131b385b0d767ff9bf60c11a4c8- Stable URL of this page
https://metraipolis.com/documento/dc_37642ce43d3f
How to cite
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation — arXiv:2608.04788 — arXiv (en), https://arxiv.org/abs/2608.04788 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_37642ce43d3f (Metraipolis archive dc_37642ce43d3f, sha256 ac454e8a5b42220ab8946b86089db0e3b86da131b385b0d767ff9bf60c11a4c8)
BibTeX
@misc{dc37642ce43d3f,
title = {Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation — arXiv:2608.04788},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.04788},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_37642ce43d3f; sha256 ac454e8a5b42220ab8946b86089db0e3b86da131b385b0d767ff9bf60c11a4c8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04788. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.04788 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:02:13.553Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).