Senteval Eval

---
name: senteval-eval
description: Evaluates the transferability and quality of universal sentence embeddings across a standardized suite of downstream tasks. It probes capabilities in sentiment classification, natural language inference, semantic textual similarity, and cross-modal image-caption retrieval using fixed hyperparameters and consistent preprocessing. Use when the user wants to benchmark on MR, CR, SUBJ, MPQA, TREC, SST-2, SST-5, SNLI, SICK-E, SICK-R, STS14, MRPC, COCO, or asks about evaluating this task. Reports accuracy, pearson.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.05449
  bibtex_key: conneau2018senteval
  confidence: high
---

# senteval-eval

> SentEval: An Evaluation Toolkit for Universal Sentence Representations — Conneau et al. (2018) (arXiv:1803.05449, 2018)

## What this evaluates

Evaluates the transferability and quality of universal sentence embeddings across a standardized suite of downstream tasks. It probes capabilities in sentiment classification, natural language inference, semantic textual similarity, and cross-modal image-caption retrieval using fixed hyperparameters and consistent preprocessing.

## Datasets

- **MR** — total 11000; splits: train (-1)
- **CR** — total 4000; splits: train (-1)
- **SUBJ** — total 10000; splits: train (-1)
- **MPQA** — total 11000; splits: train (-1)
- **TREC** — total 6000; splits: train (-1)
- **SST-2** — total 70000; splits: train (-1)
- **SST-5** — total 12000; splits: train (-1)
- **SNLI** — total 560000; splits: train (-1)
- **SICK-E** — total 10000; splits: train (-1)
- **SICK-R** — total 10000; splits: train (-1)
- **STS14** — total 4500; splits: train (-1)
- **MRPC** — total 5700; splits: train (-1)
- **COCO** — total 565000; splits: train (113000), val (5000), test (5000)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels over the total number of instances. Used for binary/multi-class classification, NLI, and paraphrase detection.
- `pearson` **(primary)** — range: [-1, 1]
  - Pearson product-moment correlation coefficient between human-annotated similarity scores and model-predicted scores (or cosine similarity for unsupervised STS).
- `spearman` — range: [-1, 1]
  - Spearman rank-order correlation coefficient between human-annotated similarity scores and model-predicted scores. Reported alongside Pearson for STS tasks.
- `recall@K` — range: [0, 1]
  - Percentage of queries where the true matching image/caption appears in the top K retrieved results. Evaluated for K in {1, 5, 10}.
- `median_rank` — range: other
  - Median of the ranks of the true matching image/caption across all queries. Lower values indicate better retrieval performance.

## Input / output format

**Input**: Single sentences for classification and STS tasks; sentence pairs for NLI, STS, and paraphrase detection; image-caption pairs for retrieval.

**Output**: Class labels (e.g., pos/neg, entailment/neutral/contradiction), continuous similarity scores in [0, 5], or ranked lists of images/captions.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type in ['classification', 'nli', 'paraphrase']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'sts':
        from scipy.stats import pearsonr, spearmanr
        p_corr, _ = pearsonr(golds, predictions)
        s_corr, _ = spearmanr(golds, predictions)
        return p_corr, s_corr
    elif task_type == 'retrieval':
        ranks = [rank_of_true_match(q) for q in queries]
        recall_k = [sum(1 for r in ranks if r <= k) / len(ranks) for k in [1, 5, 10]]
        med_r = median(ranks)
        return recall_k, med_r
```

## Common pitfalls

- Using inconsistent preprocessing or hyperparameters across models, which breaks fair comparison (SentEval enforces a fixed pipeline).
- Confusing supervised STS tasks (SICK-R, STS14) that require training a predictor, with unsupervised STS tasks (STS12-16) that only compute cosine similarity between fixed embeddings.
- Reporting only Pearson correlation for STS; the protocol requires both Pearson and Spearman, plus their average/weighted average across subtasks.

## Evidence (verbatim from paper)

> For semantic relatedness, which consists of predicting a semantic score between 0 and 5 from two input sentences, we follow the approach of Tai et al. (2015a) and learn to predict the probability distribution of relatedness scores. SentEval reports Pearson and Spearman correlation.

## Citation

```bibtex
@misc{conneau2018senteval,
  title={SentEval: An Evaluation Toolkit for Universal Sentence Representations},
  author={Conneau et al. (2018)},
  year={2018},
  note={arXiv:1803.05449}
}
```

- arXiv: 1803.05449