Metraipolis › archive › dc_16ea17b267df

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student? — arXiv:2610.31918

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.31918
License (as archived)
CC0-1.0
Archived
2026-10-26T19:47:35.576Z
SHA-256 of the archived text
ccf6f97c3222abaf436d6e5cb909cf3c84ffc1a64b0c6f55fb679ef40f73eada
Stable URL of this page
https://metraipolis.com/documento/dc_16ea17b267df

How to cite

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student? — arXiv:2610.31918 — arXiv (en), https://arxiv.org/abs/2610.31918 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_16ea17b267df (Metraipolis archive dc_16ea17b267df, sha256 ccf6f97c3222abaf436d6e5cb909cf3c84ffc1a64b0c6f55fb679ef40f73eada)
BibTeX
@misc{dc16ea17b267df,
  title = {Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student? — arXiv:2610.31918},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31918},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_16ea17b267df; sha256 ccf6f97c3222abaf436d6e5cb909cf3c84ffc1a64b0c6f55fb679ef40f73eada},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31918. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.31918 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:33.270Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).