Beerqa Eval

---
name: beerqa-eval
description: Evaluates open-domain question answering systems on their ability to retrieve and synthesize information across varying numbers of reasoning steps (single-hop to three-hop) without relying on structured metadata or predefined hop counts. Use when the user wants to benchmark on SQuAD Open, HotpotQA, BeerQA, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.12527
  bibtex_key: qi2020answering
  confidence: high
---

# beerqa-eval

> Answering Open-Domain Questions of Varying Reasoning Steps from Text — Peng Qi et al. (2020) (arXiv:2010.12527, 2020)

## What this evaluates

Evaluates open-domain question answering systems on their ability to retrieve and synthesize information across varying numbers of reasoning steps (single-hop to three-hop) without relying on structured metadata or predefined hop counts.

## Datasets

- **SQuAD Open** — total 75841; splits: train (59285), dev (8132), test (8424)
- **HotpotQA** — total 86725; splits: train (74758), dev (5989), test (5978)
- **BeerQA** — total 163096; splits: train (134043), dev (14121), test (14932); repo https://github.com/beerqa/IRRR

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer string exactly matches the gold answer string, else 0.
- `unigram F1` — range: [0, 1]
  - Token-level unigram F1 score computed between the predicted answer and the gold answer.

## Input / output format

**Input**: Question text and retrieved Wikipedia paragraphs/pages.

**Output**: Answer string.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = []
    for p, g in zip(predictions, golds):
        p_tokens = set(p.lower().split())
        g_tokens = set(g.lower().split())
        if not p_tokens or not g_tokens: continue
        prec = len(p_tokens & g_tokens) / len(p_tokens)
        rec = len(p_tokens & g_tokens) / len(g_tokens)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}
```

## Common pitfalls

- Naively merging corpora causes repeated or contradicting information, leading models to use corpus style as a shortcut or generate false answers.
- Models may fail to generalize to unseen multi-hop reasoning steps if trained only on fixed-hop datasets.
- Corpora are mapped to a newer Wikipedia dump (Aug 2020), requiring careful filtering of removed or edited pages to maintain answer validity.

## Evidence (verbatim from paper)

> For all benchmark datasets, we report standard answer exact match (EM) and unigram F1 metrics.

## Citation

```bibtex
@misc{qi2020answering,
  title={Answering Open-Domain Questions of Varying Reasoning Steps from Text},
  author={Peng Qi et al. (2020)},
  year={2020},
  note={arXiv:2010.12527}
}
```

- arXiv: 2010.12527