Multilingual Sts Eval

---
name: multilingual-sts-eval
description: Evaluates the ability of sentence embedding models to capture semantic similarity across monolingual and cross-lingual sentence pairs. It probes how well vector spaces are aligned across different languages and whether fine-tuning on English NLI/STS data generalizes to other languages. Use when the user wants to benchmark on STS 2017, or asks about evaluating this task. Reports Spearman's rank correlation (ρ).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.09813
  bibtex_key: [REDACTED]
  confidence: high
---

# multilingual-sts-eval

> Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

## What this evaluates

Evaluates the ability of sentence embedding models to capture semantic similarity across monolingual and cross-lingual sentence pairs. It probes how well vector spaces are aligned across different languages and whether fine-tuning on English NLI/STS data generalizes to other languages.

## Datasets

- **STS 2017** — total ?; splits: test (-1)

## Metrics

- `Spearman's rank correlation (ρ)` **(primary)** — range: [0, 100]
  - Computes cosine similarity between sentence embeddings, then calculates Spearman's rank correlation coefficient between the predicted similarities and the gold human-annotated scores. Reported as ρ × 100.

## Input / output format

**Input**: Pairs of sentences (monolingual or cross-lingual).

**Output**: Cosine similarity score between the two sentence embeddings.

## Scoring recipe

```python
scores = [cosine_similarity(embed(s1), embed(s2)) for s1, s2 in pairs]
rho = spearman_rank_correlation(scores, gold_scores)
return rho * 100
```

## Common pitfalls

- Using raw mBERT/XLM-R mean pooling without fine-tuning yields poor cross-lingual performance due to misaligned vector spaces.
- Confusing monolingual evaluation (Table 1) with cross-lingual evaluation (Table 2); performance drops significantly in cross-lingual settings for non-distilled models.

## Evidence (verbatim from paper)

> For the generate sentence embeddings we compute cosine similarity and, as recommended in (Reimers et al., 2016), compute the Spearman's rank correlation  $
ho$  between the computed score and the gold score.

## Citation

```bibtex
@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}
```

- arXiv: 2004.09813