Multimodal AI models trained mostly on English and Western visual data frequently hallucinate on Arabic-language images, and no benchmark tells you why it happened, only that it did. Sarab (سراب), Arabic for "mirage," is a cause-diagnostic Arabic visual hallucination benchmark: instead of one accuracy number, it traces each wrong answer to a specific root cause, a misread image, misleading text, a cultural blind spot, or reflexive guessing under an absent answer.

Five modes, five root causes

Mode	Focus	Tests
Ground	Baseline	Plain image, direct Arabic question, no context text
Sway	Specious context	A plausible but misleading Arabic caption paired with the image
False	Incorrect context	A factually wrong caption paired with the image
Clash	Cultural counter-common-sense	AI-generated Arab/Islamic cultural-norm violations, the first Arabic-native mode of its kind
Blank	Absent answer	Correct answer removed; tests unprompted "none of the above" detection

What makes it rigorous

465 images, every one fully captioned across five Arabic Cultural Visual Vocabulary categories (architecture, attire, cuisine, cultural objects, script), and every one tied to an actual test question, nothing unused
A purpose-built human review tool vets every image, caption, and distractor before it enters any test set
Real, OpenRouter-evaluated results for four models (Gemini 2.5 Flash, Gemini 2.5 Flash Lite, GPT-4o-mini, Qwen2.5-VL-72B) reported in the paper

Status

Four of eight originally scoped models are evaluated; the three Arabic-centric models (AIN, Fanar, ALLaM) are next. Every known gap is documented in the paper and repository, not smoothed over. Dataset, code, and prompts are released for reuse.

Citation

Paper in preparation. This entry will be replaced once it's published. Until then, cite the code repository:

@misc{sarab2026,
  title        = {Sarab: A Cause-Diagnostic Arabic Visual Hallucination Evaluation Benchmark},
  author       = {Alharz, Zahra and Mahyoub, Abdulrhman and Barmandah, Hassan and Alahmari, Saad Saeed},
  year         = {2026},
  howpublished = {\url{https://github.com/HasanBGit/Sarab-Benchmark}},
  note         = {Paper in preparation; citation will be updated on publication.}
}