Metraipolis › archive › dc_60c181bea221

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00283
License (as archived)
CC0-1.0
Archived
2026-11-03T18:56:58.047Z
SHA-256 of the archived text
bf2b4d3e59e7eacd488ab917b86a3c98a94f12cc42a9dab2be0ea693d5c19a89
Stable URL of this page
https://metraipolis.com/documento/dc_60c181bea221

How to cite

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283 — arXiv (en), https://arxiv.org/abs/2611.00283 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_60c181bea221 (Metraipolis archive dc_60c181bea221, sha256 bf2b4d3e59e7eacd488ab917b86a3c98a94f12cc42a9dab2be0ea693d5c19a89)
BibTeX
@misc{dc60c181bea221,
  title = {Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm — arXiv:2611.00283},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00283},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_60c181bea221; sha256 bf2b4d3e59e7eacd488ab917b86a3c98a94f12cc42a9dab2be0ea693d5c19a89},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00283. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Held-out validation loss is the standard offline signal for choosing checkpoints and comparing models, but its relation to closed-loop success on a real robot is rarely measured. We measure it for ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$ (action expert only), trained on nested subsets of 12 to 200 demonstrations of a pick-and-place task on an SO-101 arm. We score 82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearman $\rho$ from -0.23 to -0.88), but no more strongly than the number of training episodes alone, and not at all among seeds trained on the same data. Across policies the ranking is inverted ($\rho$ = +0.90 at 200 episodes): the policy with the lowest hold-out error succeeded in 3 of 110 rollouts. The most common failure, the arm never leaving its rest pose, ended 96% of rollouts for Diffusion Policy trained on untrimmed demonstrations and 47% for SmolVLA, and cannot occur in a hold-out set made of demonstrations. Among rollouts in which the arm moved, SmolVLA's hold-out action error predicts success well ($\rho$ = -0.81). For the diffusion and flow-matching policies, hold-out loss rises two- to five-fold during training while hold-out action error falls by up to 29%. Held-out metrics are a sound diagnostic of learning but no substitute for evaluation on the robot, and we recommend reporting initiation failure and success among started rollouts as separate bench outcomes.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2611.00283 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:17:07.698Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).