Metraipolis › archive › dc_e781dbacc97b
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning — arXiv:2608.21204
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.21204
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T10:31:32.843Z
- SHA-256 of the archived text
aa2050df2b22217f743ffbd562cf2a23d12e8980433a06f3f83d27abdeb41575- Stable URL of this page
https://metraipolis.com/documento/dc_e781dbacc97b
How to cite
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning — arXiv:2608.21204 — arXiv (en), https://arxiv.org/abs/2608.21204 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e781dbacc97b (Metraipolis archive dc_e781dbacc97b, sha256 aa2050df2b22217f743ffbd562cf2a23d12e8980433a06f3f83d27abdeb41575)
BibTeX
@misc{dce781dbacc97b,
title = {Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning — arXiv:2608.21204},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.21204},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e781dbacc97b; sha256 aa2050df2b22217f743ffbd562cf2a23d12e8980433a06f3f83d27abdeb41575},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21204. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.LG). Categorie arXiv: Robotics (cs.RO) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.21204 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T10:27:56.004Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).