Metraipolis › archive › dc_5e45ec181dd1

Redteaming Leading Arabic LLMs with ASAS — arXiv:2608.21985

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.21985
License (as archived)
CC0-1.0
Archived
2026-10-18T07:41:10.237Z
SHA-256 of the archived text
7a30dded1b2e9f4dfd7042a65b87db501cb03e0f9688b8a06d9b1507231b5871
Stable URL of this page
https://metraipolis.com/documento/dc_5e45ec181dd1

How to cite

Redteaming Leading Arabic LLMs with ASAS — arXiv:2608.21985 — arXiv (en), https://arxiv.org/abs/2608.21985 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5e45ec181dd1 (Metraipolis archive dc_5e45ec181dd1, sha256 7a30dded1b2e9f4dfd7042a65b87db501cb03e0f9688b8a06d9b1507231b5871)
BibTeX
@misc{dc5e45ec181dd1,
  title = {Redteaming Leading Arabic LLMs with ASAS — arXiv:2608.21985},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.21985},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5e45ec181dd1; sha256 7a30dded1b2e9f4dfd7042a65b87db501cb03e0f9688b8a06d9b1507231b5871},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21985. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As the adoption of large language models (LLMs) grows in Arabic-speaking regions, ensuring their safety and cultural alignment is increasingly critical. However, Arabic LLM safety remains underexplored, especially in adversarial evaluation settings. We introduce the Arabic Safety Index (ASAS), the first fully human-curated Arabic benchmark for redteaming LLMs. ASAS contains 801 prompts spanning 8 safety categories and 8 attack strategies, with ideal responses in Modern Standard Arabic (MSA). We conduct a redteaming evaluation across seven leading models with Arabic capabilities, including GPT-4o, Claude 3.7 Sonnet, and regional models such as ALLaM and FANAR. Human annotators rate responses using a structured 4-point safety scale, revealing that most models fail to defend against 50% of unsafe prompts. Our findings highlight major safety gaps in high-harm categories such as weapons and illicit substances, with direct and obfuscation-based attacks proving most effective. The results also show that language alignment does not readily transfer across languages, and that automated safety judges (e.g., GPT-4o) perform poorly compared to human annotators. ASAS provides a culturally grounded benchmark and redteaming protocol to drive progress in Arabic LLM safety.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.21985 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:37:52.689Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).