Squad2.0 Eval

---
name: squad2.0-eval
description: Probes a model's ability to perform extractive reading comprehension while correctly identifying when a question cannot be answered from the provided context. It forces models to distinguish between answerable and unanswerable questions, testing knowledge gap detection and resistance to semantically relevant distractors. Use when the user wants to benchmark on SQuAD 2.0, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.03822
  bibtex_key: rajpurkar2018squad2
  confidence: high
---

# squad2.0-eval

> Know What You Don't Know: Unanswerable Questions for SQuAD — Rajpurkar et al. (2018) (arXiv:1806.03822, 2018)

## What this evaluates

Probes a model's ability to perform extractive reading comprehension while correctly identifying when a question cannot be answered from the provided context. It forces models to distinguish between answerable and unanswerable questions, testing knowledge gap detection and resistance to semantically relevant distractors.

## Datasets

- **SQuAD 2.0** — total ?; splits: dev (-1), test (-1)

## Metrics

- `exact match` — range: [0, 1]
  - 1 if the predicted answer exactly matches the gold answer, 0 otherwise.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of token-level precision and recall between the predicted answer span and the gold answer span.

## Input / output format

**Input**: A context paragraph and a question.

**Output**: A predicted answer span extracted from the context, or a decision to abstain (predict unanswerable) based on a learned probability threshold.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    em, f1 = 0, 0
    for p, g in zip(preds, golds):
        if p is None:  # abstained
            continue
        em += 1 if normalize(p) == normalize(g) else 0
        f1 += token_f1(normalize(p), normalize(g))
    return em / len(golds), f1 / len(golds)
```

## Common pitfalls

- Models frequently fail to abstain on unanswerable questions, instead predicting the crowdworker-provided plausible distractor.
- The abstention threshold is tuned on the development set to maximize F1, meaning test scores are sensitive to this calibration step.
- F1 is computed at the token level, not character level, which can yield different scores than character-level exact match baselines.

## Evidence (verbatim from paper)

> Following Rajpurkar et al. (2016), we report average exact match and F1 scores. $^{3}$  The best model, DocQA + ELMo, achieves only 66.3 F1 on the test set, 23.2 points lower than the human accuracy of 89.5 F1.

## Citation

```bibtex
@misc{rajpurkar2018squad2,
  title={Know What You Don't Know: Unanswerable Questions for SQuAD},
  author={Rajpurkar et al. (2018)},
  year={2018},
  note={arXiv:1806.03822}
}
```

- arXiv: 1806.03822