Semantic Textual Similarity Eval

---
name: semantic-textual-similarity-eval
description: Evaluates a model's ability to quantify the degree of semantic similarity between pairs of sentences, including multilingual and cross-lingual contexts. It probes fine-grained semantic matching and cross-lingual generalization rather than binary paraphrase detection. Use when the user wants to benchmark on SemEval-2017 STS, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1708.00055
  bibtex_key: cer2017semeval
  confidence: high
---

# semantic-textual-similarity-eval

> SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation — Cer et al. (2017) (arXiv:1708.00055, 2017)

## What this evaluates

Evaluates a model's ability to quantify the degree of semantic similarity between pairs of sentences, including multilingual and cross-lingual contexts. It probes fine-grained semantic matching and cross-lingual generalization rather than binary paraphrase detection.

## Datasets

- **SemEval-2017 STS** — total ?; splits: test (-1)

## Metrics

- `Pearson correlation` **(primary)** — range: [0, 1]
  - Pearson correlation coefficient computed between the model's predicted similarity scores and the human-annotated reference scores for each sentence pair.

## Input / output format

**Input**: A pair of sentences (monolingual or cross-lingual).

**Output**: A real-valued similarity score, typically on a 0 to 5 ordinal scale.

## Scoring recipe

```python
import numpy as np
from scipy.stats import pearsonr

def evaluate(predictions, gold):
    # predictions, gold: list/array of floats (0-5)
    corr, _ = pearsonr(gold, predictions)
    return corr
```

## Common pitfalls

- Using Spearman correlation instead of the specified Pearson correlation, which can yield different results on this scale.
- Treating the task as binary paraphrase detection rather than fine-grained similarity scoring.
- Failing to align cross-lingual sentence pairs correctly before scoring, leading to artificially low correlations.

## Evidence (verbatim from paper)

> STS is the assessment of pairs of sentences according to their degree of semantic similarity. The task involves producing real-valued similarity scores for sentence pairs. Performance is measured by the Pearson correlation of machine scores with human judgments. The ordinal scale in Table 1 guides human annotation, ranging from 0 for no meaning overlap to 5 for meaning equivalence.

## Citation

```bibtex
@misc{cer2017semeval,
  title={SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation},
  author={Cer et al. (2017)},
  year={2017},
  note={arXiv:1708.00055}
}
```

- arXiv: 1708.00055