Metraipolis › archive › dc_4dcd5e477e56

OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development — arXiv:2608.16022

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.16022
License (as archived)
CC0-1.0
Archived
2026-10-18T19:55:24.815Z
SHA-256 of the archived text
e5b8d600b89fe8b06ead6c5f3be3ed48771b4038a58093b56fdaf64c15e086e9
Stable URL of this page
https://metraipolis.com/documento/dc_4dcd5e477e56

How to cite

OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development — arXiv:2608.16022 — arXiv (en), https://arxiv.org/abs/2608.16022 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4dcd5e477e56 (Metraipolis archive dc_4dcd5e477e56, sha256 e5b8d600b89fe8b06ead6c5f3be3ed48771b4038a58093b56fdaf64c15e086e9)
BibTeX
@misc{dc4dcd5e477e56,
  title = {OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development — arXiv:2608.16022},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.16022},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4dcd5e477e56; sha256 e5b8d600b89fe8b06ead6c5f3be3ed48771b4038a58093b56fdaf64c15e086e9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16022. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We present OPENHARMONY BENCH, an app-level coding benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS applications. Unlike function-level benchmarks, it evaluates complete app-level changes: each task requires an agent to modify a buildable ArkTS project so that a requested behavior works end to end, involving UI state, data persistence, build configuration, and platform APIs. The benchmark installs and drives the delivered application on a device to check whether the behavior is observable. It covers three input sources: natural-language feature requests (new-feature), structured scenario specifications (spec-driven), and bug descriptions (bug-fix). The benchmark contains 153 top-level tasks and 242 Feature points (F-points), where an F-point is one executable behavior check. The snapshot includes 32 new-feature tasks, 50 spec-driven tasks with 139 F-points, and 71 bug-fix tasks. The main leaderboard is scored over top-level tasks rather than independently weighted F-points. We describe the benchmark construction, statistics, and build-and-test evaluation pipeline, and evaluate DevEco Code with eight LLMs across three independent full-suite runs per configuration. Three findings emerge. First, newer generations complete more tasks than their predecessors within evaluated model-family pairs. Second, buildability is close to saturated while behavioral correctness is not: mean Final Build Success Rate is 94.77% to 100.00%, whereas mean Task Completion is 48.36% to 58.39%. Third, spec-driven tasks have the lowest Task Completion under all-checks task scoring, with no configuration exceeding 35%. The code, data, tasks, reference solutions, tests, evaluation scripts, and leaderboard are released through the official OPENHARMONY BENCH website at this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-18 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.16022 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T19:41:47.854Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).