Bucc Bitext Retrieval Eval

---
name: bucc-bitext-retrieval-eval
description: Evaluates the capability of sentence embeddings to retrieve exact translation pairs (bitexts) from large monolingual corpora across different languages. It tests the model's ability to distinguish parallel sentences from semantically similar but non-parallel ones. Use when the user wants to benchmark on BUCC bitext mining task, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.09813
  bibtex_key: reimers2020multilingual
  confidence: high
---

# bucc-bitext-retrieval-eval

> Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

## What this evaluates

Evaluates the capability of sentence embeddings to retrieve exact translation pairs (bitexts) from large monolingual corpora across different languages. It tests the model's ability to distinguish parallel sentences from semantically similar but non-parallel ones.

## Datasets

- **BUCC bitext mining task** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 100]
  - Binary classification F1 score for identifying parallel sentence pairs above a learned threshold. Threshold is optimized on the training set.

## Input / output format

**Input**: Sentence embeddings from two monolingual corpora (e.g., English and German).

**Output**: A similarity score computed via a margin function over cosine similarities and k-nearest neighbor averages, followed by a binary parallel/non-parallel decision.

## Scoring recipe

```python
def score(x, y, k=10):
    cos_xy = cosine_similarity(x, y)
    nn_x = k_nearest_neighbors(x, corpus_other_lang, k)
    nn_y = k_nearest_neighbors(y, corpus_src_lang, k)
    avg_nn = (sum(cosine_similarity(x, z) for z in nn_x) / (2*k)) + (sum(cosine_similarity(y, z) for z in nn_y) / (2*k))
    return cos_xy / avg_nn
threshold = optimize_threshold(train_scores, train_labels)
preds = [1 if score(x, y) > threshold else 0 for x, y in test_pairs]
return f1_score(test_labels, preds)
```

## Common pitfalls

- BUCC contains false negatives: Wikipedia sentences labeled non-parallel are often actually valid translations, inflating false positives for good models.
- Models optimized for semantic similarity will score non-parallel but semantically similar sentences highly, which is penalized on BUCC but correct for semantic tasks.

## Evidence (verbatim from paper)

> Performance is measured using  $F_{1}$  score. ... score(x, y) = margin(cos(x, y), sum_{z in NN_k(x)} cos(x,z)/(2k) + sum_{z in NN_k(y)} cos(y,z)/(2k)) with margin(a, b) = a/b.

## Citation

```bibtex
@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}
```

- arXiv: 2004.09813