Metraipolis › archive › dc_3ba14293ad49
Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05658
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:56:17.066Z
- SHA-256 of the archived text
cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083- Stable URL of this page
https://metraipolis.com/documento/dc_3ba14293ad49
How to cite
Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658 — arXiv (en), https://arxiv.org/abs/2609.05658 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3ba14293ad49 (Metraipolis archive dc_3ba14293ad49, sha256 cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083)
BibTeX
@misc{dc3ba14293ad49,
title = {Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05658},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3ba14293ad49; sha256 cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05658. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correctness on a single syntactic representation of an input. Such point accuracy does not reveal whether a model's correct output is stable when the same RTL behavior is written differently. This paper presents a controlled metamorphic evaluation of LLM-based SVA generation under semantics-preserving RTL transformations. Starting from the VERT dataset, we construct a quality-filtered conditional-control pool and a stratified 40-program evaluation set containing 295 assignment behaviors. We evaluate two open code models, Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite, with an identical evaluation prompt and greedy decoding. Three transformations are studied: operand reordering, deterministic identifier renaming, and redundant parenthesization. Beyond baseline and transformed accuracy, we measure conditional robustness, invariance failure, and any-flip rate, with 10,000-sample clustered bootstrap intervals at the RTL-program level. Across all six model-transformation conditions, 9.7%-27.0% of behaviors that were correct on the original RTL become incorrect after a semantics-preserving transformation. Aggregate accuracy can therefore hide substantial instability: under identifier renaming, DeepSeek-Coder-V2-Lite improves from 53.9% to 63.7% accuracy while 19.5% of its originally correct behaviors fail. Manual review of 30 sampled correct-to-wrong transitions identifies dropped path predicates, branch-polarity errors, Boolean-structure corruption, and output-contract violations. The results show that point accuracy alone is insufficient for characterizing LLM reliability in assertion generation and motivate robustness-aware evaluation for AI-assisted hardware verification. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.05658 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:39:42.799Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).