Squad V2.0 Eval

---
name: squad-v2.0-eval
description: Extends extractive QA by allowing questions that have no answer in the passage, testing the model's ability to abstain or predict a null span. It evaluates robustness against unanswerable questions. Use when the user wants to benchmark on SQuAD v2.0, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.04805
  bibtex_key: devlin2018bert
  confidence: high
---

# squad-v2.0-eval

> BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin et al. (2018) (arXiv:1810.04805, 2018)

## What this evaluates

Extends extractive QA by allowing questions that have no answer in the passage, testing the model's ability to abstain or predict a null span. It evaluates robustness against unanswerable questions.

## Datasets

- **SQuAD v2.0** — total ?; splits: train (-1), dev (-1), test (-1); HF `squad_v2`

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score, extended to handle null answers. Predicted null is compared against gold null or text spans appropriately.

## Input / output format

**Input**: Question and passage packed into a single sequence. The [CLS] token represents the null answer span.

**Output**: A start position, an end position, and a threshold comparison against a null score to decide between a text span and a null answer.

## Scoring recipe

```python
def compute_f1_v2(preds, golds):
    best_span = max(preds, key=lambda x: x.start_score + x.end_score)
    null_score = best_span.cls_score
    threshold = tune_threshold_on_dev(preds, golds)
    if best_span.score > null_score + threshold:
        pred_answer = best_span.text
    else:
        pred_answer = ""
    return squad_v2_f1_score(pred_answer, golds)
```

## Common pitfalls

- A threshold τ must be tuned on the dev set to maximize F1 when deciding between a null answer and a text span.
- The null answer is represented by the [CLS] token, and its score is compared against the best non-null span score.

## Evidence (verbatim from paper)

> The SQuAD 2.0 task extends the SQuAD 1.1 problem definition by allowing for the possibility that no short answer exists in the provided paragraph... We predict a non-null answer when ŝ_i,j > s_null + τ, where the threshold τ is selected on the dev set to maximize F1. ... We observe a +5.1 F1 improvement over the previous best system.

## Citation

```bibtex
@misc{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin et al. (2018)},
  year={2018},
  note={arXiv:1810.04805}
}
```

- arXiv: 1810.04805