Orca Eval

---
name: orca-eval
description: Evaluates Arabic language understanding across seven task clusters, including sentence classification, structured prediction, semantic similarity, NLI, QA, WSD, and topic classification. It probes models' ability to handle diverse Arabic varieties (MSA and dialects) and multiple linguistic levels from tokens to documents. Use when the user wants to benchmark on ORCA, or asks about evaluating this task. Reports ORCA score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.10758
  bibtex_key: elmadany2022orca
  confidence: high
---

# orca-eval

> ORCA: A Challenging Benchmark for Arabic Language Understanding — Elmadany et al. (2022) (arXiv:2212.10758, 2022)

## What this evaluates

Evaluates Arabic language understanding across seven task clusters, including sentence classification, structured prediction, semantic similarity, NLI, QA, WSD, and topic classification. It probes models' ability to handle diverse Arabic varieties (MSA and dialects) and multiple linguistic levels from tokens to documents.

## Datasets

- **ORCA** — total 588200; splits: train (487100), dev (46000), test (55100)

## Metrics

- `ORCA score` **(primary)** — range: [0, 1]
  - A macro-average of the performance scores across all 29 tasks and 7 task clusters, with each task weighted equally. Per-task scores are computed using standard metrics for each task type (e.g., accuracy, F1, Pearson correlation).

## Input / output format

**Input**: Varies by task cluster: single sentences for classification, sentence pairs for STS/NLI, token sequences for NER/POS, and question-context pairs for QA.

**Output**: Task-specific predictions: class labels, span boundaries, similarity scores, or generated answers.

## Scoring recipe

```python
task_scores = []
for task in all_29_tasks:
    preds, golds = get_predictions_and_labels(task)
    task_scores.append(compute_task_metric(preds, golds))
orca_score = sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Dataset sizes are capped at 50k/5k/5k for GPU-friendly evaluation, which may not reflect performance on original full-scale datasets.
- The ORCA score treats all 29 tasks equally, potentially masking performance differences across task clusters or linguistic varieties.
- Leaderboard submissions require strict metadata (parameter count, pretraining data size, fine-tuning epochs) for fair comparison.

## Evidence (verbatim from paper)

> Simple evaluation metric. We adopt a simple evaluation approach in the form of an ORCA score. The ORCA score is simply a macro-average of the different scores across all tasks and task clusters, where each task is weighted equally.

## Citation

```bibtex
@misc{elmadany2022orca,
  title={ORCA: A Challenging Benchmark for Arabic Language Understanding},
  author={Elmadany et al. (2022)},
  year={2022},
  note={arXiv:2212.10758}
}
```

- arXiv: 2212.10758