Metraipolis › archive › dc_357f04d3dca7
Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics — arXiv:2609.36608
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36608
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:57:27.853Z
- SHA-256 of the archived text
a35281906d9864c298115a4ffbbb8da5c3a89f2af25604341501950cc272e9b2- Stable URL of this page
https://metraipolis.com/documento/dc_357f04d3dca7
How to cite
Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics — arXiv:2609.36608 — arXiv (en), https://arxiv.org/abs/2609.36608 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_357f04d3dca7 (Metraipolis archive dc_357f04d3dca7, sha256 a35281906d9864c298115a4ffbbb8da5c3a89f2af25604341501950cc272e9b2)
BibTeX
@misc{dc357f04d3dca7,
title = {Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics — arXiv:2609.36608},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36608},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_357f04d3dca7; sha256 a35281906d9864c298115a4ffbbb8da5c3a89f2af25604341501950cc272e9b2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36608. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose ActFirst-OPD, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of $2.3\times$ on ALFWorld, $1.8\times$ on WebShop, and $4.9\times$ on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine benchmark-model settings. These results demonstrate that reasoning need not block acting during multi-turn agent distillation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.36608 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:27:03.405Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).