Metraipolis › archive › dc_2fdf4364765f
Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models — arXiv:2611.00896
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00896
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:44:24.287Z
- SHA-256 of the archived text
cc179eca0436c05b7b9dad0f86f45ba4174df1c84c255fa3705a27152c56afaf- Stable URL of this page
https://metraipolis.com/documento/dc_2fdf4364765f
How to cite
Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models — arXiv:2611.00896 — arXiv (en), https://arxiv.org/abs/2611.00896 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2fdf4364765f (Metraipolis archive dc_2fdf4364765f, sha256 cc179eca0436c05b7b9dad0f86f45ba4174df1c84c255fa3705a27152c56afaf)
BibTeX
@misc{dc2fdf4364765f,
title = {Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models — arXiv:2611.00896},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00896},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2fdf4364765f; sha256 cc179eca0436c05b7b9dad0f86f45ba4174df1c84c255fa3705a27152c56afaf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00896. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Simulation-based VLA evaluation often treats matching task, seed, and episode identifiers as sufficient evidence that two policies faced identical test conditions. Under specific vectorized reset and autoreset conditions, different termination timing can change subsequent initial-state assignment, so the same episode index need not mean the same test state. We present an evaluation-integrity framework for paired comparison - invariants I1-I5, provenance logging, optional scene fingerprints, and an automated checker - and a controlled LeRobot 0.6.1 LIBERO harness study with emulated controllers. Four evidence levels are kept separate: (A) mixed-horizon Sync wave-1 IDs [20, 5, 22, 7] vs [4, 5, 6, 7]; (B) homogeneous separate-job Sync control (16 cells, 0 ID/fingerprint mismatches); (C) Async+Freeze batch-size map differences (12 I3 mismatches; bs=8 incomplete; mechanism unisolated); (D) a synthetic McNemar illustration only. Controllers were emulated and no real-VLA shared-vector experiment was completed. We recommend checking declared/resolved init-state identity and provenance before paired statistics. We do not claim that published VLA leaderboards are generally invalid. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno). Categorie arXiv: Robotics (cs.RO). Pagina del lavoro: https://arxiv.org/abs/2611.00896 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:06:52.245Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).