Squad Eval

---
name: squad-eval
description: Measures a model's ability to extract precise answer spans from a given context paragraph in response to a natural language question, testing reading comprehension and span prediction. Use when the user wants to benchmark on SQuAD 1.1/2.0, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.06037
  bibtex_key: [REDACTED]
  confidence: high
---

# squad-eval

> Confidence Regularized Masked Language Modeling using Text Length — Ji et al. (2025) (arXiv:2504.06037, 2025)

## What this evaluates

Measures a model's ability to extract precise answer spans from a given context paragraph in response to a natural language question, testing reading comprehension and span prediction.

## Datasets

- **SQuAD 1.1/2.0** — total ?; splits: test (-1); HF `squad`

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score between predicted answer span and ground truth answer span, calculated as the harmonic mean of precision and recall across all examples.

## Input / output format

**Input**: Question and context paragraph tokenized together, truncated to 512 tokens.

**Output**: Predicted start and end token indices for the answer span.

## Scoring recipe

```python
def compute_f1(pred_spans, gold_spans):
    precisions, recalls = [], []
    for pred, gold in zip(pred_spans, gold_spans):
        p = len(set(pred) & set(gold)) / max(len(pred), 1)
        r = len(set(pred) & set(gold)) / max(len(gold), 1)
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        precisions.append(p)
        recalls.append(r)
    return sum(precisions) / len(precisions), sum(recalls) / len(recalls)
```

## Common pitfalls

- SQuAD 2.0 includes unanswerable questions, requiring a null prediction strategy that affects F1 calculation.
- Not averaging results over multiple random seeds as the paper reports averages of 7 trials.

## Evidence (verbatim from paper)

> We evaluated methods on the GLUE benchmark (Wang et al., 2019) and SQuAD 1.1/2.0 datasets (Rajpurkar et al., 2016, 2018). Following Devlin et al. (2019), we excluded WNLI from tasks of GLUE benchmark. We reported Matthew's correlation score for CoLA, Pearson correlations for STS-b, F1 score for SQuAD 1.1/2.0, and accuracy scores for the other tasks.

## Citation

```bibtex
@misc{ji2025confidenceregularized,
  title={Confidence Regularized Masked Language Modeling using Text Length},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.06037}
}
```

- arXiv: 2504.06037