Metraipolis › archive › dc_0ef1cf3dcbea
RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild — arXiv:2611.02497
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02497
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:28:16.841Z
- SHA-256 of the archived text
a205894a624122729fbcd3bbea171da5b5f80b7ed8c12bbaffb7225443c75128- Stable URL of this page
https://metraipolis.com/documento/dc_0ef1cf3dcbea
How to cite
RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild — arXiv:2611.02497 — arXiv (en), https://arxiv.org/abs/2611.02497 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ef1cf3dcbea (Metraipolis archive dc_0ef1cf3dcbea, sha256 a205894a624122729fbcd3bbea171da5b5f80b7ed8c12bbaffb7225443c75128)
BibTeX
@misc{dc0ef1cf3dcbea,
title = {RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild — arXiv:2611.02497},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02497},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ef1cf3dcbea; sha256 a205894a624122729fbcd3bbea171da5b5f80b7ed8c12bbaffb7225443c75128},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02497. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Robotics (cs.RO) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.02497 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:19:50.182Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).