Metraipolis › archive › dc_edece391cb6b

Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study — arXiv:2610.30946

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30946
License (as archived)
CC0-1.0
Archived
2026-10-25T08:36:43.962Z
SHA-256 of the archived text
5a60f037cbe54b04a25c352c7c4b0566c281805b982c5a77815046743ab9c3a0
Stable URL of this page
https://metraipolis.com/documento/dc_edece391cb6b

How to cite

Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study — arXiv:2610.30946 — arXiv (en), https://arxiv.org/abs/2610.30946 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edece391cb6b (Metraipolis archive dc_edece391cb6b, sha256 5a60f037cbe54b04a25c352c7c4b0566c281805b982c5a77815046743ab9c3a0)
BibTeX
@misc{dcedece391cb6b,
  title = {Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study — arXiv:2610.30946},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30946},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edece391cb6b; sha256 5a60f037cbe54b04a25c352c7c4b0566c281805b982c5a77815046743ab9c3a0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30946. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) can generate UML class diagrams from natural language requirements, but hallucinations may compromise the correctness and completeness of the resulting diagrams. Existing evaluations mostly assess the final diagram, without examining the intermediate traces of reasoning-capable LLMs, which can reveal where these hallucinations arise. We develop an empirical taxonomy of eleven hallucination sub-types organised into three dimensions: Prompt Faithfulness (deviations between requirements and reasoning), Code Faithfulness (discrepancies between reasoning and generated PlantUML code), and Reasoning Quality (problems within the reasoning). The taxonomy was derived from 60 manually annotated DeepSeek-R1 traces, applied to 120 further DeepSeek-R1 traces through a dual-LLM annotation pipeline, and examined on 20 DeepSeek-V4-Pro traces. Code Faithfulness was the most frequent dimension for DeepSeek-R1 in both datasets, whereas Prompt Faithfulness dominated for DeepSeek-V4-Pro, suggesting model-dependent failure patterns. In the manually annotated traces, reasoning-trace length showed no significant monotonic association with hallucination frequency, whereas requirement complexity was negatively associated with it, particularly for Prompt Faithfulness errors. In a mitigation experiment on 13 Golden UML ModelSet cases with three Qwen3-based models, adding Prompt Faithfulness and Reasoning Quality constraints to the generation prompt raised the pooled mean F1 score by 4.0 percentage points, whereas a post-hoc Code Faithfulness review provided no consistent further benefit. Relationship modelling remained the lowest-scoring element throughout. These results suggest that localising hallucinations within the generation process can inform targeted, prompt-based mitigation. Whether these findings generalise beyond the DeepSeek and Qwen3 model families remains to be tested.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.30946 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:27.834Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).