Orb Eval

---
name: orb-eval
description: Evaluates machine reading comprehension models across diverse linguistic phenomena such as coreference resolution, temporal logic, and causal inference. It tests generalization capabilities by applying synthetic out-of-distribution augmentations to questions, ensuring models rely on reading comprehension rather than external information retrieval. Use when the user wants to benchmark on ORB Benchmark (NewsQA, Quoref, DROP, SQuAD 1.1, SQuAD 2.0, ROPES, DuoRC, NarrativeQA), or asks about evaluating this task. Reports EM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.12598
  bibtex_key: dua2019orb
  confidence: high
---

# orb-eval

> ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension — Dua et al. (2019) (arXiv:1912.12598, 2019)

## What this evaluates

Evaluates machine reading comprehension models across diverse linguistic phenomena such as coreference resolution, temporal logic, and causal inference. It tests generalization capabilities by applying synthetic out-of-distribution augmentations to questions, ensuring models rely on reading comprehension rather than external information retrieval.

## Datasets

- **ORB Benchmark (NewsQA, Quoref, DROP, SQuAD 1.1, SQuAD 2.0, ROPES, DuoRC, NarrativeQA)** — total ?; splits: dev (-1), test (-1)

## Metrics

- `EM` **(primary)** — range: [0, 1]
  - 1.0 if the predicted answer exactly matches the gold answer (case-insensitive), 0.0 otherwise. Averaged across all instances.
- `Token F1` — range: [0, 1]
  - Harmonic mean of token-level precision and recall between predicted and gold answers. Precision = common tokens / predicted tokens. Recall = common tokens / gold tokens. Averaged across instances.

## Input / output format

**Input**: A context passage and a natural language question.

**Output**: Predicted answer span (start and end token indices in the context) or free-form text. For free-form datasets, training uses the context span with the highest ROUGE-L to the gold answer, but evaluation uses the original gold answer.

## Scoring recipe

```python
from collections import Counter
def compute_em_f1(preds, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(preds, golds):
        pred_tokens = pred.lower().split()
        gold_tokens = gold.lower().split()
        em_scores.append(1.0 if pred_tokens == gold_tokens else 0.0)
        if len(gold_tokens) == 0:
            f1_scores.append(0.0)
            continue
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        precision = num_same / len(pred_tokens) if len(pred_tokens) > 0 else 0.0
        recall = num_same / len(gold_tokens)
        f1 = (2 * precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
        f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Catastrophic forgetting occurs when sampling randomly from pooled datasets; requires uniform per-dataset sampling initially.
- Free-form answers in DuoRC/NarrativeQA are converted to ROUGE-L best spans for training labels, but evaluation uses original gold answers.
- Performance is heavily dominated by SQuAD 1.1, masking weaknesses on other datasets.

## Evidence (verbatim from paper)

> This helped improve the performance on several dataset by 3 - 4% in EM, however, there is still a lot of room for improvement on this front.

## Citation

```bibtex
@misc{dua2019orb,
  title={ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension},
  author={Dua et al. (2019)},
  year={2019},
  note={arXiv:1912.12598}
}
```

- arXiv: 1912.12598