Metraipolis › archive › dc_bf7f3bb4f92f
Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.23304
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:43:13.394Z
- SHA-256 of the archived text
fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b- Stable URL of this page
https://metraipolis.com/documento/dc_bf7f3bb4f92f
How to cite
Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304 — arXiv (en), https://arxiv.org/abs/2610.23304 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf7f3bb4f92f (Metraipolis archive dc_bf7f3bb4f92f, sha256 fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b)
BibTeX
@misc{dcbf7f3bb4f92f,
title = {Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.23304},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf7f3bb4f92f; sha256 fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23304. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI); Databases (cs.DB). Pagina del lavoro: https://arxiv.org/abs/2610.23304 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:39:10.152Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).