Gptaraeval Eval

---
name: gptaraeval-eval
description: Evaluates large language models on Arabic natural language understanding and generation across 44 tasks and over 60 datasets, covering both Modern Standard Arabic and dialectal varieties. Use when the user wants to benchmark on GPTAraEval Benchmark Suite, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14976
  bibtex_key: khondaker2023gptaraeval
  confidence: high
---

# gptaraeval-eval

> GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP — Khondaker et al. (2023) (arXiv:2305.14976, 2023)

## What this evaluates

Evaluates large language models on Arabic natural language understanding and generation across 44 tasks and over 60 datasets, covering both Modern Standard Arabic and dialectal varieties.

## Datasets

- **GPTAraEval Benchmark Suite** — total ?; splits: test (200), train (-1)

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all classes for NLU tasks. NLG tasks use task-appropriate metrics (e.g., exact match, ROUGE, BLEU) as detailed in the paper's Table 2.

## Input / output format

**Input**: Prompt containing the task instruction, optionally augmented with k few-shot examples (k ∈ {0, 3, 5, 10}) sampled from the training set, followed by the test instance.

**Output**: Free-form text response generated by the model, post-processed by stripping leading and trailing whitespace.

## Scoring recipe

```python
# For each dataset:
test_subset = random.sample(full_test_set, 200)
predictions = []
for instance in test_subset:
    prompt = build_prompt(instance, few_shot_examples)
    response = model.generate(prompt, temperature=0.0)
    predictions.append(response.strip())
if task_type == "NLU":
    metric_value = macro_f1(gold_labels=test_subset.labels, preds=predictions)
else:
    metric_value = task_specific_metric(gold=test_subset.text, preds=predictions)
```

## Common pitfalls

- Evaluation uses a fixed 200-example subset from each test set rather than the full test set, which may limit statistical power and comparability with full-benchmark scores.
- Few-shot examples are nested (k-shot includes all examples from n < k shots), which differs from standard independent sampling and may artificially inflate performance trends.
- Post-processing only removes leading/trailing whitespace, potentially leaving formatting artifacts that negatively impact exact-match or string-based NLG metrics.

## Evidence (verbatim from paper)

> For evaluation, we randomly sample a set of 200 examples from the test set of each dataset, to keep the cost manageable. We evaluate ChatGPT (gpt-3.5-turbo), which is an optimized version of GPT-3.5 series. We set the temperature to 0.0 while generating responses from ChatGPT. We compare this model with BLOOMZ (7.1B parameters)... For NLU tasks, we use macro-F1 scores and for NLG tasks we use the appropriate metric suited to each task (Table 2).

## Citation

```bibtex
@misc{khondaker2023gptaraeval,
  title={GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP},
  author={Khondaker et al. (2023)},
  year={2023},
  note={arXiv:2305.14976}
}
```

- arXiv: 2305.14976