Tatoeba Similarity Search Eval

---
name: tatoeba-similarity-search-eval
description: Evaluates cross-lingual sentence retrieval accuracy for low-resource languages by finding the most similar sentence in a target language corpus for each source sentence. It probes the alignment quality of vector spaces for languages with limited parallel data. Use when the user wants to benchmark on Tatoeba test setup (LASER), or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.09813
  bibtex_key: reimers2020multilingual
  confidence: high
---

# tatoeba-similarity-search-eval

> Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

## What this evaluates

Evaluates cross-lingual sentence retrieval accuracy for low-resource languages by finding the most similar sentence in a target language corpus for each source sentence. It probes the alignment quality of vector spaces for languages with limited parallel data.

## Datasets

- **Tatoeba test setup (LASER)** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 100]
  - Fraction of correctly retrieved parallel sentences. Computed for both directions (English to target language and target language to English) and averaged.

## Input / output format

**Input**: Sentence embeddings for English and target language sentences.

**Output**: Index of the most similar sentence in the target language corpus (or vice versa) based on cosine similarity.

## Scoring recipe

```python
correct = 0
total = 0
for direction in ['en->xx', 'xx->en']:
    for src_sent, gold_tgt_sent in pairs:
        src_emb = embed(src_sent)
        tgt_embs = [embed(t) for t in tgt_corpus]
        pred_idx = argmax(cosine_similarity(src_emb, tgt_embs))
        if tgt_corpus[pred_idx] == gold_tgt_sent:
            correct += 1
        total += 1
return (correct / total) * 100
```

## Common pitfalls

- Low-resource languages like Tagalog and Tatar were not in XLM-R's pre-training vocabulary, yet distillation still yields high accuracy, showing robustness to OOV vocabulary.
- Evaluation only covers 4 specific low-resource languages (KA, SW, TL, TT) and may not generalize to all low-resource settings.

## Evidence (verbatim from paper)

> Evaluation is done by finding for all sentences the most similar sentence in the other language using cosine similarity. Accuracy is computed for both directions (English to the other language and back).

## Citation

```bibtex
@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}
```

- arXiv: 2004.09813