Metraipolis › archive › dc_47491c6624ec
SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications — arXiv:2609.33699
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33699
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:57:05.916Z
- SHA-256 of the archived text
069d1525fc3f7a31bddc9b96945770b4a7510866df250320dbd665a726c1c3d2- Stable URL of this page
https://metraipolis.com/documento/dc_47491c6624ec
How to cite
SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications — arXiv:2609.33699 — arXiv (en), https://arxiv.org/abs/2609.33699 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_47491c6624ec (Metraipolis archive dc_47491c6624ec, sha256 069d1525fc3f7a31bddc9b96945770b4a7510866df250320dbd665a726c1c3d2)
BibTeX
@misc{dc47491c6624ec,
title = {SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications — arXiv:2609.33699},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33699},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_47491c6624ec; sha256 069d1525fc3f7a31bddc9b96945770b4a7510866df250320dbd665a726c1c3d2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33699. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Existing benchmarks for large language models (LLMs) in hardware design evaluate downstream artifacts such as generated RTL, assertions, or testbenches. When a model fails such a benchmark, the failure is ambiguous: it may have misread the specification, or it may have understood the specification and failed to write the code. We present SpecRead, a benchmark that isolates specification comprehension from generation ability. SpecRead v2.1 contains 385 questions over 10 open-source OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specifications, and spec-RTL consistency checking, plus 82 controls (41 distractor, 41 consistent-RTL). Type-4 items are built from real RTL mutations; we retain only mutations that Icarus Verilog simulation shows to change observable behavior. A with-spec vs. without-spec ablation suggests the questions require the excerpt, not training recall alone (without-spec accuracy 3/20 on the t1/t2 subset), though memorization of the source text may still help spot mutations. As an initial characterization with a small model, Ministral-3B scores 33.2% overall (128/385; macro average 39.0%): 55.2% on retrieval, 51.7% on cross-section reasoning. On the two contradiction-focused types, the verdict-plus-location measure gives 48.0% (t3) and 63.3% (t4), with a 51.2% false-positive rate on distractors and 100% on consistent-RTL controls. Layered scoring shows the model locates contradictions well (78.9-81.6% location accuracy) but scores lower on their category (43.9-49.7%). A structured "rule-table" prompting intervention lowers accuracy on every question type except t2 (tied). SpecRead is automatically scorable by deterministic checks, with gray-zone cases counted wrong under the conservative main scoring. The benchmark is regenerable for type-3 items via mutation injection, and built exclusively from public sources. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.33699 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.348Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).