Metraipolis › archive › dc_7a80aaa0ee5f

When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks — arXiv:2610.41857

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41857
License (as archived): CC0-1.0
Archived: 2026-11-02T05:05:49.477Z
SHA-256 of the archived text: 8f810300b7dfe7690dc8dde7f0f54dc902c6019c3f0ba2e28f4d387fcf5fd96d
Stable URL of this page: https://metraipolis.com/documento/dc_7a80aaa0ee5f

How to cite

When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks — arXiv:2610.41857 — arXiv (en), https://arxiv.org/abs/2610.41857 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a80aaa0ee5f (Metraipolis archive dc_7a80aaa0ee5f, sha256 8f810300b7dfe7690dc8dde7f0f54dc902c6019c3f0ba2e28f4d387fcf5fd96d)

BibTeX

@misc{dc7a80aaa0ee5f,
  title = {When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks — arXiv:2610.41857},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41857},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a80aaa0ee5f; sha256 8f810300b7dfe7690dc8dde7f0f54dc902c6019c3f0ba2e28f4d387fcf5fd96d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41857. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2610.41857 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T04:49:28.229Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).