Metraipolis › archive › dc_81dfe637d565

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration — arXiv:2608.23179

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.23179
License (as archived)
CC0-1.0
Archived
2026-10-18T07:45:15.877Z
SHA-256 of the archived text
4109bfc9fec89237b7d1956693a4e4935f05816247dd45915a4317bda1513f4e
Stable URL of this page
https://metraipolis.com/documento/dc_81dfe637d565

How to cite

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration — arXiv:2608.23179 — arXiv (en), https://arxiv.org/abs/2608.23179 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_81dfe637d565 (Metraipolis archive dc_81dfe637d565, sha256 4109bfc9fec89237b7d1956693a4e4935f05816247dd45915a4317bda1513f4e)
BibTeX
@misc{dc81dfe637d565,
  title = {NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration — arXiv:2608.23179},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.23179},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_81dfe637d565; sha256 4109bfc9fec89237b7d1956693a4e4935f05816247dd45915a4317bda1513f4e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23179. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI).
Categorie arXiv: Networking and Internet Architecture (cs.NI) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.23179 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:37:52.926Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).