Paws Eval

---
name: paws-eval
description: This benchmark evaluates a model's ability to identify paraphrases in sentence pairs that share high lexical overlap but differ in meaning due to word order and syntactic structure. It specifically probes sensitivity to non-local contextual information and adversarial word scrambling, revealing whether models rely on superficial word matching rather than true semantic understanding. Use when the user wants to benchmark on PAWS_QQP, PAWS_Wiki, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1904.01130
  bibtex_key: zhang2019paws
  confidence: high
---

# paws-eval

> PAWS: Paraphrase Adversaries from Word Scrambling — Yuan Zhang et al. (2019) (arXiv:1904.01130, 2019)

## What this evaluates

This benchmark evaluates a model's ability to identify paraphrases in sentence pairs that share high lexical overlap but differ in meaning due to word order and syntactic structure. It specifically probes sensitivity to non-local contextual information and adversarial word scrambling, revealing whether models rely on superficial word matching rather than true semantic understanding.

## Datasets

- **PAWS_QQP** — total ?; splits: dev (-1)
- **PAWS_Wiki** — total ?; splits: test (-1)

## Metrics

- `classification accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified sentence pairs out of the total. A fixed probability threshold of 0.5 is applied to model outputs before computing the ratio.
- `AUC` — range: [0, 1]
  - Area under the precision-recall curve, computed across all classification thresholds to measure the trade-off between precision and recall.

## Input / output format

**Input**: Pairs of sentences (e.g., two questions or two Wikipedia sentences) to be classified as paraphrases or non-paraphrases.

**Output**: Binary classification label (paraphrase vs. non-paraphrase) or a continuous probability score used for thresholding and AUC computation.

## Scoring recipe

```python
# Accuracy
preds = [1 if p >= 0.5 else 0 for p in probs]
acc = sum(p == g for p, g in zip(preds, gold)) / len(gold)

# AUC (Precision-Recall)
prec, rec, _ = precision_recall_curve(gold, probs)
auc = np.trapz(prec, rec)
```

## Common pitfalls

- Models frequently drop to near-chance accuracy (<40%) because they over-rely on superficial lexical overlap and ignore word order/syntax.
- PAWS_QQP evaluation is strictly on the development set, as the authors explicitly note it lacks a test set.
- AUC is computed on precision-recall curves rather than ROC curves, which is critical for correctly assessing performance on this specific task.

## Evidence (verbatim from paper)

> We use two metrics: classification accuracy and area-under-curve (AUC) scores of precision-recall curves. For all classification models, 0.5 is the threshold used to compute accuracy. We report results on testing sets for QQP and PAWS_Wiki, and on the development set for PAWS_QQP (which has no test set).

## Citation

```bibtex
@misc{zhang2019paws,
  title={PAWS: Paraphrase Adversaries from Word Scrambling},
  author={Yuan Zhang et al. (2019)},
  year={2019},
  note={arXiv:1904.01130}
}
```

- arXiv: 1904.01130