Metraipolis › archive › dc_eef20decee98

ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41168
License (as archived): CC0-1.0
Archived: 2026-11-03T07:32:42.715Z
SHA-256 of the archived text: 4f6c88c6ac88064d6ed82fd4bbad63fdb45be8383eefc04e4d43ed4092ace4dc
Stable URL of this page: https://metraipolis.com/documento/dc_eef20decee98

How to cite

ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168 — arXiv (en), https://arxiv.org/abs/2610.41168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eef20decee98 (Metraipolis archive dc_eef20decee98, sha256 4f6c88c6ac88064d6ed82fd4bbad63fdb45be8383eefc04e4d43ed4092ace4dc)

BibTeX

@misc{dceef20decee98,
  title = {ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41168},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eef20decee98; sha256 4f6c88c6ac88064d6ed82fd4bbad63fdb45be8383eefc04e4d43ed4092ace4dc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:29:33.710Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).