Metraipolis › archive › dc_56dbf5f230b9
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding — arXiv:2609.33530
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33530
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:01:31.361Z
- SHA-256 of the archived text
d56872ff83bd434ea7f8374d517b567968790db354bd62f1f4406773e09cc2af- Stable URL of this page
https://metraipolis.com/documento/dc_56dbf5f230b9
How to cite
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding — arXiv:2609.33530 — arXiv (en), https://arxiv.org/abs/2609.33530 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_56dbf5f230b9 (Metraipolis archive dc_56dbf5f230b9, sha256 d56872ff83bd434ea7f8374d517b567968790db354bd62f1f4406773e09cc2af)
BibTeX
@misc{dc56dbf5f230b9,
title = {E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding — arXiv:2609.33530},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33530},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_56dbf5f230b9; sha256 d56872ff83bd434ea7f8374d517b567968790db354bd62f1f4406773e09cc2af},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33530. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Natural Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human languages, ranging from lexical ambiguity to high-level reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models' limitations and capabilities, so optimizing models' generalization. Notably, several benchmarks contain diagnostics datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics dataset, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN dataset helps NLU designers better understand their models by doing error analysis and in-depth investigation. We used E-CONAN to investigate the performance of 9 pretrained language models and 5 LLMs. Results indicate that LLMs outperform pretrained models in world knowledge and commonsense reasoning macro-category, and underperform pretrained models in syntactic macro-category. Moreover, the hardest phenomena for all models is Reasoning, and the easiest phenomena for all pretrained models is Syntactic, and the easiest for LLMs is Lexico-Syntactic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Logic in Computer Science (cs.LO). Pagina del lavoro: https://arxiv.org/abs/2609.33530 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.683Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).