Metraipolis › archive › dc_9ccedc40348c

Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach — arXiv:2609.07290

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.07290
License (as archived)
CC0-1.0
Archived
2026-10-16T14:09:04.571Z
SHA-256 of the archived text
2b847dca7655408e3f4616315834f496d76ed5339867d1542564d319cc5c1b13
Stable URL of this page
https://metraipolis.com/documento/dc_9ccedc40348c

How to cite

Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach — arXiv:2609.07290 — arXiv (en), https://arxiv.org/abs/2609.07290 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9ccedc40348c (Metraipolis archive dc_9ccedc40348c, sha256 2b847dca7655408e3f4616315834f496d76ed5339867d1542564d319cc5c1b13)
BibTeX
@misc{dc9ccedc40348c,
  title = {Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach — arXiv:2609.07290},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.07290},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9ccedc40348c; sha256 2b847dca7655408e3f4616315834f496d76ed5339867d1542564d319cc5c1b13},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07290. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The rapid digital transformation of manufacturing, often referred to as Industry 4.0, relies on seamless interoperability between physical and software assets. A central enabler is the Asset Administration Shell (AAS), a standardized digital representation of such assets. Recent advances in large language models (LLMs) enable the generation of AAS submodels from unstructured sources such as product datasheets but raise challenges for quality assurance. In particular, unexpected errors, the lack of ground truth references, and the absence of standardized quality metrics hinder reliable adoption. In this work, we evaluate quality metrics for AI-generated AAS using a perturbation-based evaluation framework. By systematically degrading AAS generation along multiple dimensions, we assess how well different metrics reflect quality changes. Based on a dataset of 200 products from multiple manufacturers, we generate 6,400 AAS instances using GPT-4o-mini, Qwen3, and DeepSeek-R1. Our results show that metrics based on exact matching of property names and similarity-based soft matching of property values, in particular value-based recall and name-based F1 score, provide the most reliable indicators of quality degradation. Furthermore, we quantify the impact of different perturbation types and analyze differences across model families and product segments. These findings support the selection of suitable metrics, the tuning of LLM-based pipelines, and the integration of AI-generated AAS into industrial applications.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.07290 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:58:36.377Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).