Metraipolis › archive › dc_2c2bca59188d

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI — arXiv:2609.22285

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22285
License (as archived)
CC0-1.0
Archived
2026-10-09T22:21:17.472Z
SHA-256 of the archived text
f20a89a4e8e3b1b401327ff830e53b7d18d6e5cb8f13a2902be87dc5013ea4fa
Stable URL of this page
https://metraipolis.com/documento/dc_2c2bca59188d

How to cite

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI — arXiv:2609.22285 — arXiv (en), https://arxiv.org/abs/2609.22285 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c2bca59188d (Metraipolis archive dc_2c2bca59188d, sha256 f20a89a4e8e3b1b401327ff830e53b7d18d6e5cb8f13a2902be87dc5013ea4fa)
BibTeX
@misc{dc2c2bca59188d,
  title = {ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI — arXiv:2609.22285},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22285},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c2bca59188d; sha256 f20a89a4e8e3b1b401327ff830e53b7d18d6e5cb8f13a2902be87dc5013ea4fa},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22285. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the model already knows, performance gains cannot be cleanly attributed to new learning rather than pre-existing knowledge. This matters most for knowledge-intensive, task-light (KHTL) robot deployments - pharmaceutical dispensing, hazardous-material handling, facility-specific protocols, where the physical task is simple but the governing rules are proprietary and safety-critical, and where extensive live testing is costly or unsafe. We introduce ORDER (Ontology-driven Decision-making for Embodied Reasoning), a benchmark built on a fictitious world: a 342,069-token synthetic corpus defining a self-consistent physics that cannot appear in any model's pre-training data. ORDER pairs a 500-question knowledge test (ORDER-BENCH) with a harder compositional task, ORDER-SPATIAL: ordering objects for safe manipulation across both familiar and entirely novel scenes. GPT-4.1 without adaptation scores below chance on ORDER-SPATIAL (Kendall's tau = 0.441), showing its priors actively conflict with the invented physics. After continual pre-training, small models improve substantially on both familiar and novel scenes alike evidence of genuine world-model induction rather than memorization. We then carry this through to a robot pipeline: models that answer the knowledge test well often cannot produce valid, executable plans without a further skill-adaptation stage, after which small, fully offline models outperform GPT-4.1 even when GPT-4.1 is given retrieval access to the same rules (Kendall's tau = 0.848 vs. 0.606), on a full perception-to-execution loop demonstrated on a simulated iiwa7 arm with human-in-the-loop correction. Throughout, ORDER-SPATIAL performance, not knowledge-test accuracy is what predicts real plan quality.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-22 in cs.RO).
Categorie arXiv: Robotics (cs.RO) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.22285 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T21:59:59.408Z dalla pagina https://arxiv.org/catchup/cs.RO/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).