Mmteb Retrieval Eval

---
name: mmteb-retrieval-eval
description: Evaluates the dense retrieval capability of multilingual embedding models across multiple languages and document-query pairs. It measures how well compact models can match or exceed larger baselines on standardized multilingual retrieval benchmarks. Use when the user wants to benchmark on MTEB (Multilingual), or asks about evaluating this task. Reports MMTEB(Retrieval).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.14274
  bibtex_key: [REDACTED]
  confidence: medium
---

# mmteb-retrieval-eval

> Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters — Tu et al. (2025) (arXiv:2510.14274, 2025)

## What this evaluates

Evaluates the dense retrieval capability of multilingual embedding models across multiple languages and document-query pairs. It measures how well compact models can match or exceed larger baselines on standardized multilingual retrieval benchmarks.

## Datasets

- **MTEB (Multilingual)** — total ?; splits: test (-1)

## Metrics

- `MMTEB(Retrieval)` **(primary)** — range: percent
  - Average retrieval score across the multilingual MTEB benchmark tasks. Typically computed as nDCG@10 per task and averaged across all languages/tasks.

## Input / output format

**Input**: Query-document pairs for dense retrieval evaluation.

**Output**: Not specified in the provided text.

## Scoring recipe

```python
scores = []
for task in mmteb_retrieval_tasks:
    scores.append(compute_ndcg_at_10(predictions, gold))
final_score = sum(scores) / len(scores)
return final_score
```

## Common pitfalls

- Context length limitations of some baseline models (e.g., multilingual-e5-large-instruct supports ~500 tokens) can artificially cap retrieval performance.
- Data scale alone does not guarantee continuous gains; hard negative mining and task diversity are critical for further improvement.

## Evidence (verbatim from paper)

> | Model | #Params | Supported Context Length | Emb. Dim. | MMTEB(Retrieval) |
| --- | --- | --- | --- | --- |
| MGTE | 305M | 8k | 768 | 56.50 |
| bge-m3 | 568M | 8k | 4096 | 55.59 |
| multilingual-e5-large-instruct | 560M | 514 | 1024 | 57.12 |
| snowflake-l | 568M | 8k | 1024 | 58.36 |
| Our | 305M | 8k | 768 | 60.56 |
| gte-Qwen2-1.5B-instruct | 1.5B | 32k | 1536 | 60.78 |
| SFR-Embedding-Mistral | 7B | 4k | 4096 | 59.44 |
| gte-Qwen2-7B-instruct | 7B | 32k | 3584 | 60.08

*Table 1: Performance of models in the MTEB (Multilingual) retrieval tasks.*

## Citation

```bibtex
@misc{tu2025retrofitting,
  title={Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters},
  author={Tu et al. (2025)},
  year={2025},
  note={arXiv:2510.14274}
}
```

- arXiv: 2510.14274