Metraipolis › archive › dc_afb7e70d709e

Learning Robotic World Models with Semantic and Visual Feedback — arXiv:2611.07850

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07850
License (as archived)
CC0-1.0
Archived
2026-11-06T14:53:52.582Z
SHA-256 of the archived text
ef14363d9b79bc6266147458b252ef117a9033d8d025316a0a33a28ffa312a07
Stable URL of this page
https://metraipolis.com/documento/dc_afb7e70d709e

How to cite

Learning Robotic World Models with Semantic and Visual Feedback — arXiv:2611.07850 — arXiv (en), https://arxiv.org/abs/2611.07850 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_afb7e70d709e (Metraipolis archive dc_afb7e70d709e, sha256 ef14363d9b79bc6266147458b252ef117a9033d8d025316a0a33a28ffa312a07)
BibTeX
@misc{dcafb7e70d709e,
  title = {Learning Robotic World Models with Semantic and Visual Feedback — arXiv:2611.07850},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07850},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_afb7e70d709e; sha256 ef14363d9b79bc6266147458b252ef117a9033d8d025316a0a33a28ffa312a07},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07850. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Robot world models predict action-conditioned future visual states to support robotic planning and decision making. However, visual reconstruction does not fully capture task progress, semantic evaluation alone provides limited constraints on fine-grained visual errors, and homogeneous predictions limit comparative feedback. We present RoboImagine, a framework for post-training autoregressive robotic world models through candidate-future comparisons with semantic and visual feedback. Diverse search promotes diversity among candidate trajectories under shared visual observations, action sequences, and task instructions. A frozen multimodal evaluator provides task-relevant semantic preferences through pairwise video comparisons in both presentation orders, complemented by pixel- and perceptual-level visual consistency feedback that constrains local structural and appearance errors. These signals drive policy optimization through group-relative advantages. Fixed initial context and branch-specific rolling histories support long-sequence generation with bounded context and consistent conditioning between candidate generation and policy optimization. Experiments with separately trained models on RT-1, BridgeV2, and RoboCasa cover real-world and simulated manipulation. On RT-1, under matched initialization, update counts, and evaluation protocols, RoboImagine improves PSNR by 1.26 dB and reduces LPIPS by 11.93% relative to continued supervised fine-tuning. Ablations show that diverse search improves prediction quality and that combined semantic and visual feedback yields higher external multimodal evaluation scores than visual feedback alone while maintaining comparable visual fidelity. Rolling histories further improve long-horizon prediction. These results support improving visual fidelity, task relevance, and long-horizon prediction through candidate comparison and complementary feedback.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.07850 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:25:42.634Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).