Metraipolis › archive › dc_7ecb781f64ef

DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.05168
License (as archived): CC0-1.0
Archived: 2026-11-04T18:57:41.535Z
SHA-256 of the archived text: f6a59117dd9029a6530ea0cdff1f877a712f114baf82cac71122722488baa4af
Stable URL of this page: https://metraipolis.com/documento/dc_7ecb781f64ef

How to cite

DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168 — arXiv (en), https://arxiv.org/abs/2611.05168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7ecb781f64ef (Metraipolis archive dc_7ecb781f64ef, sha256 f6a59117dd9029a6530ea0cdff1f877a712f114baf82cac71122722488baa4af)

BibTeX

@misc{dc7ecb781f64ef,
  title = {DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05168},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7ecb781f64ef; sha256 f6a59117dd9029a6530ea0cdff1f877a712f114baf82cac71122722488baa4af},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used in software-engineering workflows, but code-generation benchmarks largely focus on general-purpose languages. Domain-specific languages (DSLs) have distinct syntax, semantics, and validation requirements, making results from individual DSL benchmarks difficult to compare. We present DSL-Bench, an executable benchmark and analysis framework for cross-DSL code generation. Starting from 13,045 DSL-related papers, LLM-assisted open card sorting identifies 18 representative application domains. For each domain, we select an evaluable DSL and provide a syntax checker and task-level semantic oracle. Evaluating 7 LLMs on 18 DSLs, we observe mean syntax and semantic pass rates of 31.37% and 11.64%, respectively, revealing a substantial gap between syntax acceptance and semantic correctness. A variance decomposition attributes 76.99% of semantic pass-rate variation to differences among DSLs and 7.86% to differences among models. Among four DSL-level factors, Knowledge Demand has the strongest negative association with semantic correctness (r = -0.508). We evaluate prompt engineering across all 18 DSLs and study five improvement methods on five targeted DSLs. Prompt engineering yields uneven gains; RAG-Refinement, the strongest targeted method, improves semantic correctness by 16.29 percentage points on average over matched baselines. We further derive capability profiles and DSL-Profiler summaries for interpreting data-scarce DSLs. These results highlight toolchain-specific semantic bottlenecks and the need to evaluate DSL code generation beyond syntax acceptance.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.05168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:35:52.156Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).