Metraipolis › archive › dc_165ce78efdba
Closed-loop evaluation of LLM agents for embedded software development — arXiv:2610.11447
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11447
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T05:57:49.104Z
- SHA-256 of the archived text
378a8c70af4f694cf1261f31c55c8f84a2d0638957014c9d7a7b4c7e74a9e341- Stable URL of this page
https://metraipolis.com/documento/dc_165ce78efdba
How to cite
Closed-loop evaluation of LLM agents for embedded software development — arXiv:2610.11447 — arXiv (en), https://arxiv.org/abs/2610.11447 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_165ce78efdba (Metraipolis archive dc_165ce78efdba, sha256 378a8c70af4f694cf1261f31c55c8f84a2d0638957014c9d7a7b4c7e74a9e341)
BibTeX
@misc{dc165ce78efdba,
title = {Closed-loop evaluation of LLM agents for embedded software development — arXiv:2610.11447},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11447},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_165ce78efdba; sha256 378a8c70af4f694cf1261f31c55c8f84a2d0638957014c9d7a7b4c7e74a9e341},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11447. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding agents are emerging. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.11447 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:01.690Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).