Fanar20 Benchmarks Eval

---
name: fanar20-benchmarks-eval
description: Evaluates bilingual language understanding, reasoning, and instruction-following capabilities of generative AI models. It probes general knowledge, commonsense reasoning, and culturally aligned Arabic comprehension across multiple standard and custom benchmarks. Use when the user wants to benchmark on English Benchmarks (MMLU, HellaSwag, ARC-Challenge, PIQA, Winogrande), OALL v1, or asks about evaluating this task. Reports English Avg., Arabic Avg..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.16397
  bibtex_key: fanarteam2026fanar20
  confidence: medium
---

# fanar20-benchmarks-eval

> Fanar 2.0: Arabic Generative AI Stack — FANAR TEAM et al. (2026) (arXiv:2603.16397, 2026)

## What this evaluates

Evaluates bilingual language understanding, reasoning, and instruction-following capabilities of generative AI models. It probes general knowledge, commonsense reasoning, and culturally aligned Arabic comprehension across multiple standard and custom benchmarks.

## Datasets

- **English Benchmarks (MMLU, HellaSwag, ARC-Challenge, PIQA, Winogrande)** — total ?; splits: test (-1)
- **OALL v1** — total ?; splits: test (-1)

## Metrics

- `English Avg.` **(primary)** — range: percent
  - Arithmetic mean of accuracy scores across MMLU, HellaSwag, ARC-Challenge, PIQA, and Winogrande.
- `Arabic Avg.` **(primary)** — range: percent
  - Arithmetic mean of accuracy scores across the OALL v1 benchmark suite.

## Input / output format

**Input**: Multiple-choice questions, factual QA prompts, and open-ended instruction-following prompts in English and Arabic.

**Output**: Model-generated text responses, including selected options for multiple-choice tasks or free-form answers/reasoning traces for open-ended tasks.

## Scoring recipe

```python
def compute_english_avg(predictions, golds):
    scores = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    return sum(scores) / len(scores) * 100

def compute_arabic_avg(predictions, golds):
    scores = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    return sum(scores) / len(scores) * 100

# For open-ended generation (MT-Bench, Alpaca, 50+ domains):
# Score = automated_judge_model(prompt, response).score
```

## Common pitfalls

- English and Arabic averages are computed over different benchmark suites, so direct cross-lingual comparison of absolute scores is invalid.
- Open-ended generation evaluation relies on automated closed-source LLM judges rather than human raters, which may introduce scoring bias.
- The exact composition and weighting of the 'composite benchmark score' used for in-loop monitoring are not fully disclosed.

## Evidence (verbatim from paper)

> English Avg. and Arabic Avg. are reported as the average across several English benchmarks, e.g., MMLU, HellaSwag, ARC-Challenge, PIQA, Winogrande; and the OALL v1 benchmark suite average, respectively. The suite covers instruction-following (translated MT-Bench [21], Alpaca benchmarks with custom judging prompts), broad general capabilities across 50+ domains, and multi-turn conversational fluency—all evaluated on open-ended generation using strong closed-source models as automated judges.

## Citation

```bibtex
@misc{fanarteam2026fanar20,
  title={Fanar 2.0: Arabic Generative AI Stack},
  author={FANAR TEAM et al. (2026)},
  year={2026},
  note={arXiv:2603.16397}
}
```

- arXiv: 2603.16397