Metraipolis › archive › dc_bf7f3bb4f92f

Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.23304
License (as archived)
CC0-1.0
Archived
2026-10-19T08:43:13.394Z
SHA-256 of the archived text
fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b
Stable URL of this page
https://metraipolis.com/documento/dc_bf7f3bb4f92f

How to cite

Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304 — arXiv (en), https://arxiv.org/abs/2610.23304 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf7f3bb4f92f (Metraipolis archive dc_bf7f3bb4f92f, sha256 fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b)
BibTeX
@misc{dcbf7f3bb4f92f,
  title = {Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs — arXiv:2610.23304},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23304},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf7f3bb4f92f; sha256 fe48e590472ca862fd8db62c4d52c8060b0d1689fc97690acf848a502568306b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23304. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI); Databases (cs.DB).
Pagina del lavoro: https://arxiv.org/abs/2610.23304 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.152Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).