Quality First Arabic LLM Leaderboard

QIMMA validates the benchmarks before evaluating the model to ensure that the reported scores reflect the LLM’s authentic Arabic proficiency.

If you’ve been following the Arabic LLM ratings, you’ve probably noticed that tensions are rising. The number of benchmarks and leaderboards is expanding rapidly, but are we actually measuring what we think we’re measuring?

We built QIMMA قمّة (meaning “peak” in Arabic) to systematically answer that question. Rather than simply aggregating existing Arabic benchmarks and running the model on top of them, we applied a rigorous quality validation pipeline before performing the evaluation. What we discovered was sobering. Even the widely used and well-regarded Arabic benchmarks contain systemic quality issues that can quietly undermine evaluation results.

In this article, we will explain what QIMMA is, how we built it, the issues we found, and how the model ranks after cleaning.

🔍 Problem: Arabic NLP evaluation is fragmented and untested

Although Arabic is spoken by more than 400 million people across diverse dialects and cultural backgrounds, the landscape of Arabic NLP evaluation remains fragmented. Several important issues motivate this work.

Translation issues. Many of the Arabic benchmarks are translations from English. This results in a change in the distribution. Benchmark data is less representative of natural usage of Arabic, as questions that feel natural in English can be awkward or culturally out of place in Arabic.

Quality verification has not been carried out. Even native Arabic benchmarks are often released without rigorous quality checks. Annotation discrepancies, inaccurate gold answers, encoding errors, and cultural bias in ground truth labels are all documented in established resources.

Reproducibility gap. Evaluation scripts and per-sample output are rarely publicly available, making it difficult to audit results or build on previous work.

Coverage fragmentation. Existing leaderboards cover isolated tasks and narrow areas, making it difficult to evaluate the overall model.

To explain where QIMMA stands compared to existing platforms:

QIMMA is the only platform that combines all five characteristics: open source, primarily native Arabic content, systematic quality verification, code evaluation, and inference output per published sample.

⛰ What does QIMMA have?

QIMMA combines 109 subsets from 14 source benchmarks into a unified evaluation suite of more than 52,000 samples across seven domains.

Several things stand out in this design.

99% native Arabic content. The only exception is code evaluation, which is essentially language independent. The first Arabic leaderboard with code evaluation. QIMMA integrates Arabic-adapted versions of HumanEval+ and MBPP+, allowing you to assess your coding ability with Arabic problem statements. Diversity of domains and tasks. QIMMA assesses real-world competency areas such as education, governance, healthcare, creative expression, and software development.

🔬 Quality verification pipeline

This is the heart of QIMMA’s methodology. We applied a multi-stage validation pipeline to all samples of all benchmarks before running a single model.

Stage 1: Automatic multi-model evaluation

Each sample was independently evaluated by two state-of-the-art LLMs.

Qwen3-235B-A22B-DeepSeek Instructions-V3-671B

We selected two models with strong Arabic features, but their training data has different compositions, making their combined decisions more robust than either alone.

Each model scores samples against a 10-point rubric using a binary score (0 or 1) for each criterion.

If any model scores less than 7/10, the sample is excluded. Samples that both models agree on removal are immediately removed. However, if only one model flags the sample, proceed to stage 2 human review.

Stage 2: Human annotation and review

Flagged samples are reviewed by native Arabic speakers who are culturally and dialectally familiar. The human annotator will make the final decision on:

Cultural background and regional differences Dialect nuances Subjective interpretation Subtle quality issues Automatic evaluation may miss

For culturally sensitive content, multiple points of view are considered, as “correctness” can be very different in different Arab regions.

⚠️ What we found: Systemic quality issues

The pipeline revealed recurring quality issues across benchmarks. This is not an isolated error, but a systematic pattern that reflects a gap in the way the benchmark was originally constructed.

Look at the numbers

Classification of problems found

💻 Code Benchmarking: A different kind of quality work

Benchmarking the code required another intervention. Rather than discarding the sample, we refined the Arabic problem statement in the Arabic adaptations of 3LM’s HumanEval+ and MBPP+, leaving the task identifier, reference solution, and test suite completely unchanged.

The revision rate was amazing.

Changes fall into five categories:

Linguistic improvements: Normalized towards natural Modern Standard Arabic and a consistent imperative style Improved clarity: Fixed ambiguous instructions and unclear constraints Normalized consistency: Standardized formatting of mathematical terminology, punctuation, and examples Structural fixes: Fixed broken triple-quoted strings, indentation errors, and broken text fragments Semantic improvements: Clarified whether scopes were inclusive or exclusive and maintained the intent of the task

⚙️ Evaluation setup

Evaluation framework

QIMMA uses LightEval, EvalPlus, and FannOrFlop as evaluation frameworks, chosen for their consistency, multilingual community adoption, and reproducibility.

Metrics by task type

prompt template

QIMMA standardizes question-based prompts using six template types:

All prompts are in Arabic. For MizanQA and ArabCulture, benchmark-specific system prompts from the original paper are saved.

🏆 Leaderboard Results

Actual results as of April 2026. Covers the top 10 rated models. Visit our live leaderboard for current rankings.

Scale does not guarantee best performance. The top 10 ranges from models with 32B to 397B parameters, with several medium models outperforming large models in certain domains. Arabic-specific models lead cultural and linguistic tasks. Jais-2-70B-Chat ranks highest in ArabicMMLU and ArabCulture, while Karnak ranks first in 3LM STEM and ArabLegalQA. Coding remains the most difficult area for Arabic-specific models. The top HumanEval+ and MBPP+ scores belong to multilingual models, with Qwen3.5-397B leading both.

Relationship between size and performance

Across the leaderboard (46 models), a clear but imperfect correlation between size and performance emerges. However, there are some interesting exceptions.

Arabic-specific models often outperform size-matched multilingual models Instruction-adjusted models consistently outperform base models, with the exception of Qwen3 Some small Arabic-specific models (Fanar-1-9B, ALLaM-7B) outperform larger multilingual models in certain domains

🌟 QIMMA Difference

The characteristic properties of QIMMA can be summarized as follows.

🔗 Resources

🔖 Quote

@misc{alqadi2026arabicbenchmarksreliableqimmas, title={Are Arabic benchmarks reliable? QIMMA’s quality-first approach to LLM evaluation}, authors={Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai, Hamza Alobeidli, Maitha Alhammadi, Shaikha Alsuwaidi, Omar Alkaabi, Basma El Amel Boussaha, Hakim Hacid}, year={2026}, eprint={2604.03395}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2604.03395}, }