Metraipolis › archive › dc_ec025ab09f9e

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects — arXiv:2611.10802

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.10802
License (as archived)
CC0-1.0
Archived
2026-11-10T18:42:50.762Z
SHA-256 of the archived text
b5b834f1707006d4a241c9f05e3bddc3c16754f92ae5cefcfe21ea04d41e4ee7
Stable URL of this page
https://metraipolis.com/documento/dc_ec025ab09f9e

How to cite

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects — arXiv:2611.10802 — arXiv (en), https://arxiv.org/abs/2611.10802 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec025ab09f9e (Metraipolis archive dc_ec025ab09f9e, sha256 b5b834f1707006d4a241c9f05e3bddc3c16754f92ae5cefcfe21ea04d41e4ee7)
BibTeX
@misc{dcec025ab09f9e,
  title = {Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects — arXiv:2611.10802},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.10802},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec025ab09f9e; sha256 b5b834f1707006d4a241c9f05e3bddc3c16754f92ae5cefcfe21ea04d41e4ee7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10802. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How does Jev compare with other systems on Arabic tasks, and how does its performance change between Modern Standard Arabic (MSA) and dialects? We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. The Arabic-focused frozen suite includes English controls in three parallel resources, which we also remove in a separate descriptive comparison. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. Removing the recorded English controls, repeated cross-task items or both leaves the full system order unchanged. Relative performance and gains between Jev versions vary across task families. In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. Paired contrasts with Jev 1.13.0, Gemma-4-31B and Jais-2-8B show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.10802 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.607Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).