Metraipolis › archive › dc_9386772829b6

Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy — arXiv:2609.37749

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37749
License (as archived)
CC0-1.0
Archived
2026-10-07T18:37:47.332Z
SHA-256 of the archived text
2d1c0bc812207a05e1ac1be26c2a82b8afc44121a301b5b4de67639adb8f7bd2
Stable URL of this page
https://metraipolis.com/documento/dc_9386772829b6

How to cite

Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy — arXiv:2609.37749 — arXiv (en), https://arxiv.org/abs/2609.37749 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9386772829b6 (Metraipolis archive dc_9386772829b6, sha256 2d1c0bc812207a05e1ac1be26c2a82b8afc44121a301b5b4de67639adb8f7bd2)
BibTeX
@misc{dc9386772829b6,
  title = {Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy — arXiv:2609.37749},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37749},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9386772829b6; sha256 2d1c0bc812207a05e1ac1be26c2a82b8afc44121a301b5b4de67639adb8f7bd2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37749. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The increasing importance of Information Retrieval (IR) in managing large datasets has highlighted significant limitations in traditional keyword-based search systems. Context-aware chat-based search methods, such as Retrieval Augmented Generation (RAG), have recently emerged, but their evaluation compared to keyword-based systems often relies on subjective user feedback. A rigorous, quantitative comparison between these paradigms remains lacking. This work introduces a novel, preliminary framework to quantitatively assess IR accuracy of search systems that produce different output formats, such as lists and messages. It focuses on two key aspects: the ranking accuracy for keyword-based systems and the completeness of retrieved information for semantic chat-based systems. Our approach enables semi-automatic comparisons of semantic and keyword-based methods using interchangeable equivalence classes tailored to domain-specific contexts (e.g., companies or problems). We validate the framework through an industrial case study, demonstrating statistically significant improvements in context-aware search over keyword-based methods, supported by analyses including the Mann-Whitney U-Test. With its adaptable design, the proposed framework provides a strong foundation for objectively assessing keyword-based and semantic chat-based search methods.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-30 in cs.IR).
Categorie arXiv: Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2609.37749 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T18:04:00.989Z dalla pagina https://arxiv.org/catchup/cs.IR/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).