Metraipolis › archive › dc_b070fba2104f

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs — arXiv:2611.02587

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02587
License (as archived): CC0-1.0
Archived: 2026-11-03T10:58:29.603Z
SHA-256 of the archived text: 4a17e23e4d703bcb6e14a91fa993302133aa453d4944ad1cd02d8617635c01a8
Stable URL of this page: https://metraipolis.com/documento/dc_b070fba2104f

How to cite

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs — arXiv:2611.02587 — arXiv (en), https://arxiv.org/abs/2611.02587 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b070fba2104f (Metraipolis archive dc_b070fba2104f, sha256 4a17e23e4d703bcb6e14a91fa993302133aa453d4944ad1cd02d8617635c01a8)

BibTeX

@misc{dcb070fba2104f,
  title = {They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs — arXiv:2611.02587},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02587},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b070fba2104f; sha256 4a17e23e4d703bcb6e14a91fa993302133aa453d4944ad1cd02d8617635c01a8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02587. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models are increasingly consulted for religious knowledge, including the recall of specific Hadith (recorded sayings and actions of the Prophet Muhammad, peace be upon him). Unlike open-ended generation, Hadith recall is a high-stakes, verifiable task: a reference (collection, chapter, and Hadith number) has exactly one correct text, and any deviation from it constitutes a fabrication that can be mistaken for authentic scripture. This study introduces a bilingual (English/Arabic) benchmark that probes five open-weight, locally-deployable language models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B) across all six canonical Sunni Hadith collections (Sahih al-Bukhari, Sahih Muslim, Sunan Abi Da'ud, Jami' al-Tirmidhi, Sunan an-Nasa'i, Sunan Ibn Majah), sampling 500 Hadith per collection (3,000 per model per language, 30,000 responses total). Each model is given only the bibliographic reference and asked to reproduce the exact Hadith text, with no retrieval augmentation, and is scored on lexical fidelity (BLEU) and semantic fidelity (multilingual cosine similarity), with refusals flagged separately. Across all five models and both languages, 0% of 30,000 responses achieve both high lexical and high semantic fidelity, i.e., an exact, verifiable match to the source text. Arabic responses fare uniformly worse than English. These findings suggest that current open-weight LLMs, used without retrieval grounding, are unreliable and potentially dangerous sources for exact religious-text recall, and that deployment for Islamic scholarship should mandate retrieval-augmented generation over a verified corpus rather than parametric memory. The complete code, prompts, evaluation pipeline, and scored outputs are publicly available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02587 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.657Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).