Paws X Eval

---
name: paws-x-eval
description: PAWS-X evaluates a model's ability to identify paraphrases across multiple languages, specifically probing sensitivity to word order and syntactic structure under conditions of high lexical overlap. It measures how well models generalize cross-lingually when trained on machine-translated data versus zero-shot settings. Use when the user wants to benchmark on PAWS-X, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.11828
  bibtex_key: yang2019pawsx
  confidence: high
---

# paws-x-eval

> PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification — Yang et al. (2019) (arXiv:1908.11828, 2019)

## What this evaluates

PAWS-X evaluates a model's ability to identify paraphrases across multiple languages, specifically probing sensitivity to word order and syntactic structure under conditions of high lexical overlap. It measures how well models generalize cross-lingually when trained on machine-translated data versus zero-shot settings.

## Datasets

- **PAWS-X** — total 23659; splits: train (-1), dev (-1), test (-1); repo https://github.com/google-research-datasets/paws

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified sentence pairs. Computed by comparing predicted labels against gold labels.
- `AUC-PR` — range: percent
  - Area under the precision-recall curve. For BERT, computed using probability scores for the positive class. For BOW and ESIM, computed using cosine similarity scores with a 0.5 threshold.

## Input / output format

**Input**: A pair of sentences (a1, a2) in a single language (English, French, Spanish, German, Chinese, Japanese, or Korean).

**Output**: Binary classification label: 'match' (paraphrase) or 'not_MATCH' (non-paraphrase).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, model_scores=None):
    # predictions, gold_labels: list of 'match'/'not_MATCH'
    accuracy = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels) * 100
    
    auc_pr = -1
    if model_scores is not None:
        gold_binary = [1 if g == 'match' else 0 for g in gold_labels]
        prec, rec, _ = precision_recall_curve(gold_binary, model_scores)
        auc_pr = auc(rec, prec)
    return accuracy, auc_pr
```

## Common pitfalls

- Models may achieve high accuracy by exploiting translation artifacts or shared entity names rather than true cross-lingual structural understanding.
- Zero-shot evaluation on non-English languages is highly sensitive to the quality of the machine translation system used to generate training data, leading to performance gaps between Indo-European and CJK languages.
- Bag-of-words and simple similarity baselines fail because the dataset deliberately maximizes lexical overlap while varying word order, requiring explicit sensitivity to syntax.

## Evidence (verbatim from paper)

> We use two metrics: classification accuracy and area-under-curve scores of precision-recall curves (AUC-PR). For BERT, probability scores for the positive class is used to compute AUC-PR. For BOW and ESIM a cosine threshold of 0.5 is used to compute accuracy. In all experiments, the best model checkpoint is chosen based on accuracy on development sets and report results on testing sets.

## Citation

```bibtex
@misc{yang2019pawsx,
  title={PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification},
  author={Yang et al. (2019)},
  year={2019},
  note={arXiv:1908.11828}
}
```

- arXiv: 1908.11828