Metraipolis › archive › dc_0dc49a2c8a86

ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41168
License (as archived)
CC0-1.0
Archived
2026-11-02T07:40:33.953Z
SHA-256 of the archived text
66aa9f5bc0e69986bda2dc73a97c94ef8927945649d16ef043384ca50fcc708c
Stable URL of this page
https://metraipolis.com/documento/dc_0dc49a2c8a86

How to cite

ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168 — arXiv (en), https://arxiv.org/abs/2610.41168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0dc49a2c8a86 (Metraipolis archive dc_0dc49a2c8a86, sha256 66aa9f5bc0e69986bda2dc73a97c94ef8927945649d16ef043384ca50fcc708c)
BibTeX
@misc{dc0dc49a2c8a86,
  title = {ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration — arXiv:2610.41168},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41168},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0dc49a2c8a86; sha256 66aa9f5bc0e69986bda2dc73a97c94ef8927945649d16ef043384ca50fcc708c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:37:00.563Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).