Metraipolis › archive › dc_db8cf3d96f7d

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective — arXiv:2609.15387

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15387
License (as archived)
CC0-1.0
Archived
2026-10-15T22:53:47.515Z
SHA-256 of the archived text
c03f9c56c1e71b5ba2ad9cf8dbe377764b9d4048c84c493f553b80254f35d399
Stable URL of this page
https://metraipolis.com/documento/dc_db8cf3d96f7d

How to cite

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective — arXiv:2609.15387 — arXiv (en), https://arxiv.org/abs/2609.15387 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db8cf3d96f7d (Metraipolis archive dc_db8cf3d96f7d, sha256 c03f9c56c1e71b5ba2ad9cf8dbe377764b9d4048c84c493f553b80254f35d399)
BibTeX
@misc{dcdb8cf3d96f7d,
  title = {WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective — arXiv:2609.15387},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15387},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db8cf3d96f7d; sha256 c03f9c56c1e71b5ba2ad9cf8dbe377764b9d4048c84c493f553b80254f35d399},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15387. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3\% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.15387 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.848Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).