Metraipolis › archive › dc_f245bb4f4c64
DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05168
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:43:14.837Z
- SHA-256 of the archived text
43cfeeafd89f7dadea36349bb51be484880d8931529feea42f74e03330720637- Stable URL of this page
https://metraipolis.com/documento/dc_f245bb4f4c64
How to cite
DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168 — arXiv (en), https://arxiv.org/abs/2611.05168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f245bb4f4c64 (Metraipolis archive dc_f245bb4f4c64, sha256 43cfeeafd89f7dadea36349bb51be484880d8931529feea42f74e03330720637)
BibTeX
@misc{dcf245bb4f4c64,
title = {DSL-Bench: Evaluating and Explaining LLM Code Generation Across Domain-Specific Languages — arXiv:2611.05168},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05168},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f245bb4f4c64; sha256 43cfeeafd89f7dadea36349bb51be484880d8931529feea42f74e03330720637},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly used in software-engineering workflows, but code-generation benchmarks largely focus on general-purpose languages. Domain-specific languages (DSLs) have distinct syntax, semantics, and validation requirements, making results from individual DSL benchmarks difficult to compare. We present DSL-Bench, an executable benchmark and analysis framework for cross-DSL code generation. Starting from 13,045 DSL-related papers, LLM-assisted open card sorting identifies 18 representative application domains. For each domain, we select an evaluable DSL and provide a syntax checker and task-level semantic oracle. Evaluating 7 LLMs on 18 DSLs, we observe mean syntax and semantic pass rates of 31.37% and 11.64%, respectively, revealing a substantial gap between syntax acceptance and semantic correctness. A variance decomposition attributes 76.99% of semantic pass-rate variation to differences among DSLs and 7.86% to differences among models. Among four DSL-level factors, Knowledge Demand has the strongest negative association with semantic correctness (r = -0.508). We evaluate prompt engineering across all 18 DSLs and study five improvement methods on five targeted DSLs. Prompt engineering yields uneven gains; RAG-Refinement, the strongest targeted method, improves semantic correctness by 16.29 percentage points on average over matched baselines. We further derive capability profiles and DSL-Profiler summaries for interpreting data-scarce DSLs. These results highlight toolchain-specific semantic bottlenecks and the need to evaluate DSL code generation beyond syntax acceptance. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.05168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T12:30:02.142Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).