Qqp Paraphrase Generation Eval

---
name: qqp-paraphrase-generation-eval
description: Evaluates a model's ability to generate semantically equivalent paraphrase sentences from an input question, measuring lexical and semantic overlap with ground truth references. The benchmark probes sentence-level semantic understanding and generative fluency in a question-paraphrase setting. Use when the user wants to benchmark on Quora Question Pairs (QQP), or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.13149
  bibtex_key: patro2019revisiting
  confidence: high
---

# qqp-paraphrase-generation-eval

> Revisiting Paraphrase Question Generator using Pairwise Discriminator — Patro et al. (2019) (arXiv:1912.13149, 2019)

## What this evaluates

Evaluates a model's ability to generate semantically equivalent paraphrase sentences from an input question, measuring lexical and semantic overlap with ground truth references. The benchmark probes sentence-level semantic understanding and generative fluency in a question-paraphrase setting.

## Datasets

- **Quora Question Pairs (QQP)** — total 400000; splits: train (50000), train (100000), val (5000), test (30000); repo https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs

## Metrics

- `BLEU` **(primary)** — range: [0, 1]
  - Standard n-gram precision metric with a brevity penalty to penalize overly short generations. Typically reported as BLEU-1 (unigram precision).
- `METEOR` — range: [0, 1]
  - Harmonic mean of unigram precision and recall, incorporating stemming and synonymy matching to better capture semantic similarity than exact n-gram overlap.
- `TER` — range: [0, 1]
  - Translation Error Rate; calculated as the number of edit operations (insertions, deletions, substitutions, shifts) required to match the reference, divided by the reference length.
- `ROUGE` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation; measures n-gram overlap between the generated and reference text, focusing on recall.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation; uses TF-IDF weighting to measure how well the generated text matches the consensus of multiple reference texts.

## Input / output format

**Input**: Original question text (string)

**Output**: Generated paraphrase question text (string)

## Scoring recipe

```python
def compute_metrics(predictions, references):
    bleu = nltk.translate.bleu_score.sentence_bleu(references, pred, weights=(1,0,0,0))
    meteor = nltk.translate.meteor_score.single_meteor_score(ref, pred)
    ter = ter_score(pred, ref)  # edit distance ops / ref length
    rouge = nltk.translate.rouge_score.rouge_l(ref, pred)['fmeasure']
    cider = nltk.translate.cider_score.cider_score(ref, pred)[0]
    return {'BLEU': bleu, 'METEOR': meteor, 'TER': ter, 'ROUGE': rouge, 'CIDEr': cider}
```

## Common pitfalls

- The dataset split is non-standard: 50k/100k training pairs, 5k validation, and 30k test, following specific prior works rather than the official Quora split.
- Metrics are reported as absolute scores and percentage improvements over baselines, but exact tokenization/preprocessing details for BLEU/METEOR are not specified, which can cause score variance across implementations.

## Evidence (verbatim from paper)

> We observe that the score EDLPS method performs best across all scores (BLEU, ROUGE, METEOR, and CIDEr).

## Citation

```bibtex
@misc{patro2019revisiting,
  title={Revisiting Paraphrase Question Generator using Pairwise Discriminator},
  author={Patro et al. (2019)},
  year={2019},
  note={arXiv:1912.13149}
}
```

- arXiv: 1912.13149