Pearl Eval

---
name: pearl-eval
description: Evaluates large vision-language models' ability to understand and generate culturally-aware Arabic content across multiple reasoning-centric question types. It probes hypothesis formation, comparative analysis, chronological reasoning, and explicit cultural grounding in both closed-form and open-ended multimodal tasks. Use when the user wants to benchmark on PeARL, or asks about evaluating this task. Reports relaxed-match accuracy (ACC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21979
  bibtex_key: alwajih2025pearl
  confidence: high
---

# pearl-eval

> Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset — Alwajih et al. (2025) (arXiv:2505.21979, 2025)

## What this evaluates

Evaluates large vision-language models' ability to understand and generate culturally-aware Arabic content across multiple reasoning-centric question types. It probes hypothesis formation, comparative analysis, chronological reasoning, and explicit cultural grounding in both closed-form and open-ended multimodal tasks.

## Datasets

- **PeARL** — total ?; splits: test (-1); repo https://github.com/UBC-NLP/pearl

## Metrics

- `relaxed-match accuracy (ACC)` **(primary)** — range: [0, 1]
  - Binary score (1 if semantically equivalent to gold, 0 otherwise) averaged over closed-form questions. Permits synonyms, paraphrases, or minor lexical variations.
- `Overall Score` — range: [0.2, 5.0]
  - Weighted sum of four dimensions scored 1-5: 0.4*Correctness + 0.2*Coherence + 0.2*Detail + 0.2*Fluency.
- `Cultural Awareness Score (CAS)` — range: [0, 1]
  - Binary metric (0/1) indicating explicitly whether the candidate response mentions culturally-specific elements required by the reference answer.

## Input / output format

**Input**: Multimodal prompts containing images and Arabic text questions, varying between closed-form (multiple-choice, True/False) and open-ended types (cause-and-effect, comparative analysis, scenario completion, etc.).

**Output**: Model-generated Arabic text responses. For closed-form, a direct answer; for open-ended, a free-form explanation or completion.

## Scoring recipe

```python
def score_closed(pred, gold):
    return 1 if is_semantically_equivalent(pred, gold) else 0

def score_open(pred, gold):
    c = judge_score(pred, gold, 'correctness')
    coh = judge_score(pred, gold, 'coherence')
    d = judge_score(pred, gold, 'detail')
    f = judge_score(pred, gold, 'fluency')
    return 0.4*c + 0.2*coh + 0.2*d + 0.2*f

def score_cas(pred, gold):
    return 1 if mentions_cultural_elements(pred, gold) else 0
```

## Common pitfalls

- The LVLM judge's scoring may be biased by its own training data or cultural background, despite being instruction-tuned for Arabic.
- Relaxed-match accuracy relies on semantic equivalence judgments which can vary between judges or models, potentially inflating scores for paraphrased but factually incorrect answers.
- The weighted Overall Score heavily emphasizes correctness (0.4), which might overshadow coherence or fluency in open-ended evaluations.

## Evidence (verbatim from paper)

> For closed-form questions (e.g., multiple-choice and True/False), we utilize a relaxed-match accuracy (ACC) metric. Here, the judge assesses semantic equivalence between candidate responses and gold-standard answers, permitting synonyms, paraphrases, or minor lexical variations. Each response is assigned a binary correctness score (1 for correct, 0 for incorrect), aggregated into an overall accuracy. For open-ended questions... the judge evaluates responses using a comprehensive structured rubric capturing four critical dimensions: correctness, coherence, detail, and fluency. Each dimension is scored individually on a scale from 1 to 5, with an aggregated, weighted Overall Score calculated as follows: Overall Score = 0.4 Correctness + 0.2 Coherence + 0.2 Detail + 0.2 Fluency. We also follow Burda-Lassen et al. (2025) in employing a Cultural Awareness Score (CAS). CAS is a binary metric (0/1) indicating explicitly whether the candidate response mentions culturally-specific elements required by the reference answer...

## Citation

```bibtex
@misc{alwajih2025pearl,
  title={Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset},
  author={Alwajih et al. (2025)},
  year={2025},
  note={arXiv:2505.21979}
}
```

- arXiv: 2505.21979