Squad V1.1 Eval

---
name: squad-v1.1-eval
description: Measures extractive question answering capability by requiring the model to identify a text span in a passage that answers a given question. It tests precise token-level span prediction and contextual understanding. Use when the user wants to benchmark on SQuAD v1.1, or asks about evaluating this task. Reports exact-match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.04805
  bibtex_key: devlin2018bert
  confidence: high
---

# squad-v1.1-eval

> BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin et al. (2018) (arXiv:1810.04805, 2018)

## What this evaluates

Measures extractive question answering capability by requiring the model to identify a text span in a passage that answers a given question. It tests precise token-level span prediction and contextual understanding.

## Datasets

- **SQuAD v1.1** — total 100000; splits: train (-1), dev (-1), test (-1); HF `squad`

## Metrics

- `exact-match (EM)` **(primary)** — range: [0, 1]
  - Percentage of questions where the predicted answer string exactly matches any of the ground truth answer strings.
- `F1` — range: [0, 1]
  - Token-level F1 score between the predicted answer and the ground truth answer, averaged over all questions.

## Input / output format

**Input**: Question and passage packed into a single sequence with distinct segment embeddings (A for question, B for passage).

**Output**: A start position and an end position for the answer span within the passage.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    em = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1s = [token_f1(p, g) for p, g in zip(preds, golds)]
    return em, sum(f1s)/len(f1s)
```

## Common pitfalls

- The model predicts start/end logits over all tokens; the best span is chosen by maximizing start+end scores, not by independently picking max start and max end.
- Ensembling multiple models with different seeds/checkpoints significantly boosts performance compared to single models.

## Evidence (verbatim from paper)

> The Stanford Question Answering Dataset (SQuAD v1.1) is a collection of 100k crowdsourced question/answer pairs... The score of a candidate span from position i to position j is defined as S·T_i + E·T_j, and the maximum scoring span where j ≥ i is used as a prediction. ... Table 2: SQuAD 1.1 results. ... EM | F1

## Citation

```bibtex
@misc{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin et al. (2018)},
  year={2018},
  note={arXiv:1810.04805}
}
```

- arXiv: 1810.04805