Metraipolis › archive › dc_3ba14293ad49

Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.05658
License (as archived)
CC0-1.0
Archived
2026-10-16T13:56:17.066Z
SHA-256 of the archived text
cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083
Stable URL of this page
https://metraipolis.com/documento/dc_3ba14293ad49

How to cite

Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658 — arXiv (en), https://arxiv.org/abs/2609.05658 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3ba14293ad49 (Metraipolis archive dc_3ba14293ad49, sha256 cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083)
BibTeX
@misc{dc3ba14293ad49,
  title = {Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations — arXiv:2609.05658},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.05658},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3ba14293ad49; sha256 cd9d8bcf54272776d3aa3bb378d6540f8ba39ccb8e336f2c128b6a1e9ea0a083},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05658. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correctness on a single syntactic representation of an input. Such point accuracy does not reveal whether a model's correct output is stable when the same RTL behavior is written differently. This paper presents a controlled metamorphic evaluation of LLM-based SVA generation under semantics-preserving RTL transformations. Starting from the VERT dataset, we construct a quality-filtered conditional-control pool and a stratified 40-program evaluation set containing 295 assignment behaviors. We evaluate two open code models, Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite, with an identical evaluation prompt and greedy decoding. Three transformations are studied: operand reordering, deterministic identifier renaming, and redundant parenthesization. Beyond baseline and transformed accuracy, we measure conditional robustness, invariance failure, and any-flip rate, with 10,000-sample clustered bootstrap intervals at the RTL-program level. Across all six model-transformation conditions, 9.7%-27.0% of behaviors that were correct on the original RTL become incorrect after a semantics-preserving transformation. Aggregate accuracy can therefore hide substantial instability: under identifier renaming, DeepSeek-Coder-V2-Lite improves from 53.9% to 63.7% accuracy while 19.5% of its originally correct behaviors fail. Manual review of 30 sampled correct-to-wrong transitions identifies dropped path predicates, branch-polarity errors, Boolean-structure corruption, and output-contract violations. The results show that point accuracy alone is insufficient for characterizing LLM reliability in assertion generation and motivate robustness-aware evaluation for AI-assisted hardware verification.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.05658 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.799Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).