Metraipolis › archive › dc_d0e8078fa339

Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol — arXiv:2610.21741

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21741
License (as archived)
CC0-1.0
Archived
2026-10-16T10:40:01.186Z
SHA-256 of the archived text
99a83e51d965be9ef4dc01dd24be8b6b26103cb9bd39c109a27b2a04a30e1622
Stable URL of this page
https://metraipolis.com/documento/dc_d0e8078fa339

How to cite

Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol — arXiv:2610.21741 — arXiv (en), https://arxiv.org/abs/2610.21741 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d0e8078fa339 (Metraipolis archive dc_d0e8078fa339, sha256 99a83e51d965be9ef4dc01dd24be8b6b26103cb9bd39c109a27b2a04a30e1622)
BibTeX
@misc{dcd0e8078fa339,
  title = {Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol — arXiv:2610.21741},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21741},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d0e8078fa339; sha256 99a83e51d965be9ef4dc01dd24be8b6b26103cb9bd39c109a27b2a04a30e1622},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21741. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Closed-loop evaluation is widely regarded as a more faithful way to assess end-to-end (E2E) driving models than open-loop evaluation, and Bench2Drive, built on the CARLA simulator, has become the dominant benchmark for it. We show that its evaluation protocol itself contains pitfalls that can misrepresent the models scored on it. Bench2Drive's comfort metric, adapted from nuPlan, has several implementation errors and a segmentation scheme that rewards standing still, so it mostly measures how long a vehicle stays stationary, correlating with stationary time at r = 0.97 across eight models. We propose a corrected, distance-based metric that cuts that correlation to r = 0.06. Under it, the strong driving-score/comfort tradeoff the original metric reports is no longer detectable once models are compared at matched driving speed, which suggests that comfort trades against how fast a model drives rather than against how well it drives. We also uncover two pitfalls in how the harness supplies navigation signals: the default controller can replace a model's own steering with an angle toward the route target point, and the target point supplied at evaluation time can differ substantially from what a model saw in training. We demonstrate both on UniAD and VAD, where their effects differ sharply, and argue that future closed-loop evaluations should check for them. We will release all fixes as standalone code, so corrected comfort scores can be obtained from logged trajectories without re-running closed-loop simulation.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2610.21741 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:52:17.666Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).