Metraipolis › archive › dc_bca2d4db39c0

GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents — arXiv:2610.08959

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08959
License (as archived)
CC0-1.0
Archived
2026-10-08T11:10:24.050Z
SHA-256 of the archived text
18667adfa6096b0b4f03bc2da06c3be5e8717147c937b4ea9e318f4827f7134a
Stable URL of this page
https://metraipolis.com/documento/dc_bca2d4db39c0

How to cite

GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents — arXiv:2610.08959 — arXiv (en), https://arxiv.org/abs/2610.08959 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bca2d4db39c0 (Metraipolis archive dc_bca2d4db39c0, sha256 18667adfa6096b0b4f03bc2da06c3be5e8717147c937b4ea9e318f4827f7134a)
BibTeX
@misc{dcbca2d4db39c0,
  title = {GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents — arXiv:2610.08959},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08959},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bca2d4db39c0; sha256 18667adfa6096b0b4f03bc2da06c3be5e8717147c937b4ea9e318f4827f7134a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08959. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation post-trains large language model agents by supplying dense, step-level guidance from a teacher policy when the reinforcement-learning reward is sparse and arrives only once per trajectory. Existing instantiations allocate this guidance by the size of the teacher-student divergence at each step, on the single-turn intuition that a large disagreement marks a mistake worth correcting. Once decisions chain over many turns, that rule misfires, since an early drift enters every later context both policies condition on, leaving the teacher consistent with the drifted trajectory instead of flagging its cause, while interchangeable steps register large but outcome-irrelevant divergences. We demonstrate this on an agentic benchmark, where distilling the highest-divergence steps brings no consistent benefit over random selection. To this end, we introduce GraphOPD, the first method to bring graph-based structural augmentation into on-policy distillation for agent capabilities. It reads which steps enabled which later ones from the environment's own record of state changes, immune to the drift that corrupts the teacher-student gap, organizes them into a dependency graph, scores each step by a random-walk stationary distribution over it, and fuses that structural credit with the divergence signal into a trajectory-relative mask concentrating supervision on each rollout's highest-aptitude steps. Across three model scales and eleven baselines on ALFWorld, WebShop, and SearchQA, GraphOPD shows competitive performance throughout, improving over the strongest baseline by up to +5.8 pp. An executed-replay audit further shows that this structural credit score tracks true causal impact far above chance, that both fused signals are independently necessary, and that the same signal transfers to out-of-domain tool-integrated reasoning.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.08959 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.033Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).