Low Resource Embedding Eval

---
name: low-resource-embedding-eval
description: Evaluates sentence embedding quality for low-resource languages trained on synthetic triplet data. It probes cross-lingual semantic similarity and information retrieval capabilities by benchmarking against human-annotated and unsupervised baselines without requiring target-language training data. Use when the user wants to benchmark on Ousidhoum STS/STR, MTEB Retrieval (Low-Resource Subset), or asks about evaluating this task. Reports Spearman's correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01732
  bibtex_key: basoz2026bootstrapping
  confidence: high
---

# low-resource-embedding-eval

> Bootstrapping Embeddings for Low Resource Languages — Basoz et al. (2026) (arXiv:2603.01732, 2026)

## What this evaluates

Evaluates sentence embedding quality for low-resource languages trained on synthetic triplet data. It probes cross-lingual semantic similarity and information retrieval capabilities by benchmarking against human-annotated and unsupervised baselines without requiring target-language training data.

## Datasets

- **Ousidhoum STS/STR** — total ?; splits: test (-1)
- **MTEB Retrieval (Low-Resource Subset)** — total ?; splits: test (-1)

## Metrics

- `Spearman's correlation` **(primary)** — range: [-1, 1]
  - Spearman rank correlation coefficient between predicted similarity scores and ground-truth similarity ratings. Measures the monotonic relationship between predictions and human/anchor labels.
- `Recall@10` — range: [0, 1]
  - Fraction of queries where the relevant document appears in the top 10 retrieved results.

## Input / output format

**Input**: Sentence pairs for STS tasks; query-document pairs for retrieval tasks, all in target low-resource languages.

**Output**: Continuous similarity score for STS; ranked list of document IDs with scores for retrieval.

## Scoring recipe

```python
def compute_metrics(pred_scores, gold_scores, queries, retrieved_ids, gold_doc_ids):
    # STS
    spearman_corr = scipy.stats.spearmanr(pred_scores, gold_scores).correlation
    # Retrieval
    recall_at_10 = sum(1 for q, g in zip(queries, gold_doc_ids) if g in retrieved_ids[q][:10]) / len(queries)
    # Aggregate
    return spearman_corr, recall_at_10
# Report mean ± std over 4 random seeds
```

## Common pitfalls

- Evaluating zero-shot cross-lingual transfer (trained on English, tested on target) as if it were direct target-language training.
- Ignoring seed variance; the protocol requires reporting mean ± standard deviation over four random seeds.
- Assuming synthetic triplet generation requires an LLM fine-tuned on the target language; the protocol uses cross-lingual adaptation (XL-LoRA) without target-language training data.

## Evidence (verbatim from paper)

> We evaluate performance using both STS/STR tasks from Ousidhoum et al. (2024) and a subset of retrieval tasks from MTEB Muennighoff et al. (2022) specifically focusing on low resource languages. Table 1: Embedding performance on STS tasks (Spearman’s correlation). Figure 3: Retrieval results across multiple benchmarks. Results are averaged across backbones (XLM-R and mmBERT) and across languages. Metric is recall@10.

## Citation

```bibtex
@misc{basoz2026bootstrapping,
  title={Bootstrapping Embeddings for Low Resource Languages},
  author={Basoz et al. (2026)},
  year={2026},
  note={arXiv:2603.01732}
}
```

- arXiv: 2603.01732