Mrqa 2019 Shared Task Eval

---
name: mrqa-2019-shared-task-eval
description: Evaluates out-of-domain generalization in extractive reading comprehension by testing models on held-out datasets from diverse domains (crowdsourced, synthetic, domain experts, Wikipedia, education, etc.) that were not seen during training. Use when the user wants to benchmark on MRQA 2019 Shared Task, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.09753
  bibtex_key: fisch2019mrqa
  confidence: high
---

# mrqa-2019-shared-task-eval

> MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension — Fisch et al. (2019) (arXiv:1910.09753, 2019)

## What this evaluates

Evaluates out-of-domain generalization in extractive reading comprehension by testing models on held-out datasets from diverse domains (crowdsourced, synthetic, domain experts, Wikipedia, education, etc.) that were not seen during training.

## Datasets

- **MRQA 2019 Shared Task** — total ?; splits: Split I (-1), Split II (-1), Split III (-1); repo https://github.com/mrqa/MRQA-Shared-Task-2019

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Standard extractive QA F1 score, computed as the harmonic mean of exact-match precision and recall over predicted answer spans versus gold answer spans in the context.

## Input / output format

**Input**: A context passage and a natural language question.

**Output**: A text span extracted from the context, corresponding to the start and end token indices of the answer.

## Scoring recipe

```python
def compute_macro_f1(predictions, golds):
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = set(pred.split())
        gold_tokens = set(gold.split())
        if not gold_tokens:
            f1_scores.append(0.0)
            continue
        overlap = len(pred_tokens & gold_tokens)
        prec = overlap / len(pred_tokens) if pred_tokens else 0.0
        rec = overlap / len(gold_tokens)
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models are ranked on the macro-averaged F1 across 12 held-out test datasets (Split II + III), not on in-domain performance (Split I).
- The task strictly requires extractive spans; models generating abstractive answers are penalized or disqualified.
- Data sampling restrictions: participants were only allowed to use the provided training data from six datasets, disallowing external QA data.

## Evidence (verbatim from paper)

> The teams are ranked by the F1 scores on the hidden testing portions of the 12 datasets (Split II and III in Section 3.1). Table 3 lists the macro-averaged F1 scores of all the submissions on both the development and testing portions of the MRQA dataset.

## Citation

```bibtex
@misc{fisch2019mrqa,
  title={MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension},
  author={Fisch et al. (2019)},
  year={2019},
  note={arXiv:1910.09753}
}
```

- arXiv: 1910.09753