Qrcd Eval

---
name: qrcd-eval
description: Evaluates machine reading comprehension on a low-resource religious domain (Qur'an). It probes a model's ability to extract precise answer spans from Arabic text given a question, testing both exact matching and partial semantic/token overlap. Use when the user wants to benchmark on QRCD, or asks about evaluating this task. Reports pRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.06025
  bibtex_key: premasiri2022dtw
  confidence: high
---

# qrcd-eval

> DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain — Premasiri et al. (2022) (arXiv:2205.06025, 2022)

## What this evaluates

Evaluates machine reading comprehension on a low-resource religious domain (Qur'an). It probes a model's ability to extract precise answer spans from Arabic text given a question, testing both exact matching and partial semantic/token overlap.

## Datasets

- **QRCD** — total ?; splits: dev (-1), test (-1); repo https://github.com/DamithDR/QuestionAnswering

## Metrics

- `pRR` **(primary)** — range: [0, 1]
  - Partial Reciprocal Rank. For each query, rank predictions by confidence. Score is 1/rank if the prediction partially matches any gold answer, else 0. Averaged over all queries.
- `EM` — range: [0, 1]
  - Exact Match. Binary score: 1 if the top predicted answer exactly matches at least one gold answer, 0 otherwise.
- `F1@1` — range: [0, 1]
  - Token-level F1 score computed between the top predicted answer and the best-matching gold answer.

## Input / output format

**Input**: Arabic context (Qur'anic verse or paragraph) paired with a question.

**Output**: A single text span extracted from the context as the predicted answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    prr, em, f1 = 0.0, 0.0, 0.0
    for pred, gold_list in zip(predictions, golds):
        # pRR: 1/rank of first partial match
        for i, p in enumerate(pred, 1):
            if partial_match(p, gold_list):
                prr += 1.0 / i
                break
        # EM: exact match on top prediction
        em += 1.0 if exact_match(pred[0], gold_list) else 0.0
        # F1@1: token F1 on top prediction vs best gold
        f1 += token_f1(pred[0], best_matching_gold(pred[0], gold_list))
    n = len(predictions)
    return prr/n, em/n, f1/n
```

## Common pitfalls

- Metrics are computed exclusively on the top-ranked prediction, not on ensemble outputs.
- pRR uses partial matching (token/semantic overlap) rather than strict exact match, so minor formatting differences still score points.
- Transfer learning from SOQUAL only improved AraELECTRA-discriminator; applying it to other models did not yield gains.

## Evidence (verbatim from paper)

> As advised by the task organisers, we used partial Reciprocal Rank (pRR) score to measure the model performance. It is a variant of the traditional Reciprocal Rank evaluation metric that considers partial matching. We also report Exact Match (EM), and F1@1 in the results tables, which are evaluation metrics applied only to the top predicted answer. The EM metric is a binary measure that rewards a system only if the top predicted answer matches exactly one of the gold answers. In comparison, the F1@1 metric measures the token overlap between the top predicted answer and the best matching gold answer.

## Citation

```bibtex
@misc{premasiri2022dtw,
  title={DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain},
  author={Premasiri et al. (2022)},
  year={2022},
  note={arXiv:2205.06025}
}
```

- arXiv: 2205.06025