Metraipolis › archive › dc_e58c8aeeff42

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.00283
License (as archived): CC0-1.0
Archived: 2026-11-03T11:43:49.105Z
SHA-256 of the archived text: 63e040021f2d3695c525cbae24614de93cc3c97432dcc062bd29e974cf683850
Stable URL of this page: https://metraipolis.com/documento/dc_e58c8aeeff42

How to cite

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283 — arXiv (en), https://arxiv.org/abs/2611.00283 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e58c8aeeff42 (Metraipolis archive dc_e58c8aeeff42, sha256 63e040021f2d3695c525cbae24614de93cc3c97432dcc062bd29e974cf683850)

BibTeX

@misc{dce58c8aeeff42,
  title = {Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00283},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e58c8aeeff42; sha256 63e040021f2d3695c525cbae24614de93cc3c97432dcc062bd29e974cf683850},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00283. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Held-out validation loss is the standard offline signal for choosing checkpoints and comparing models, but its relation to closed-loop success on a real robot is rarely measured. We measure it for ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$ (action expert only), trained on nested subsets of 12 to 200 demonstrations of a pick-and-place task on an SO-101 arm. We score 82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearman $\rho$ from -0.23 to -0.88), but no more strongly than the number of training episodes alone, and not at all among seeds trained on the same data. Across policies the ranking is inverted ($\rho$ = +0.90 at 200 episodes): the policy with the lowest hold-out error succeeded in 3 of 110 rollouts. The most common failure, the arm never leaving its rest pose, ended 96% of rollouts for Diffusion Policy trained on untrimmed demonstrations and 47% for SmolVLA, and cannot occur in a hold-out set made of demonstrations. Among rollouts in which the arm moved, SmolVLA's hold-out action error predicts success well ($\rho$ = -0.81). For the diffusion and flow-matching policies, hold-out loss rises two- to five-fold during training while hold-out action error falls by up to 29%. Held-out metrics are a sound diagnostic of learning but no substitute for evaluation on the robot, and we recommend reporting initiation failure and success among started rollouts as separate bench outcomes.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2611.00283 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:06:52.228Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).