Mmarco Passage Ranking Eval

---
name: mmarco-passage-ranking-eval
description: Evaluates multilingual passage retrieval models on translated versions of the MS MARCO dataset and the Mr. TyDi dataset. It probes the ability of dense retrieval and reranking models to handle cross-lingual and zero-shot retrieval scenarios, as well as the impact of translation quality on retrieval effectiveness. Use when the user wants to benchmark on mMARCO, Mr. TyDi, or asks about evaluating this task. Reports MRR@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.13897
  bibtex_key: bonifacio2021mmarco
  confidence: high
---

# mmarco-passage-ranking-eval

> mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset — Bonifacio et al. (2021) (arXiv:2108.13897, 2021)

## What this evaluates

Evaluates multilingual passage retrieval models on translated versions of the MS MARCO dataset and the Mr. TyDi dataset. It probes the ability of dense retrieval and reranking models to handle cross-lingual and zero-shot retrieval scenarios, as well as the impact of translation quality on retrieval effectiveness.

## Datasets

- **mMARCO** — total ?; splits: development (-1); repo https://github.com/unicamp-dl/mMARCO
- **Mr. TyDi** — total ?; splits: test (-1)

## Metrics

- `MRR@10` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank of the first relevant document within the top 10 retrieved passages. Calculated as the average of 1/rank for each query where rank is the position of the first relevant result.
- `Recall@1000` — range: [0, 1]
  - Fraction of all relevant passages retrieved within the top 1000 results. Calculated as the size of the intersection between retrieved and relevant sets divided by the size of the relevant set.
- `Recall@100` — range: [0, 1]
  - Fraction of all relevant passages retrieved within the top 100 results. Calculated as the size of the intersection between retrieved and relevant sets divided by the size of the relevant set.

## Input / output format

**Input**: Query and passage text in a specific language.

**Output**: Ranked list of passages for each query.

## Scoring recipe

```python
def compute_mrr_at_k(predictions, gold, k=10):
    scores = []
    for pred_list, gold_set in zip(predictions, gold):
        for rank, doc_id in enumerate(pred_list[:k], 1):
            if doc_id in gold_set:
                scores.append(1.0 / rank)
                break
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)

def compute_recall_at_k(predictions, gold, k=1000):
    scores = []
    for pred_list, gold_set in zip(predictions, gold):
        retrieved = set(pred_list[:k])
        scores.append(len(retrieved & gold_set) / len(gold_set))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Queries and passages are translated independently, causing lexical mismatches that penalize lexical models like BM25.
- Zero-shot evaluation tests models on languages completely absent from the fine-tuning data, which can yield lower performance than in-domain evaluation.
- The correlation between translation quality (BLEU) and retrieval effectiveness (MRR@10) is weak (R^2 ≈ 0.33), so high BLEU does not guarantee better retrieval.

## Evidence (verbatim from paper)

> We report MRR@10, which is the official metric of the MS MARCO passage dataset, as well as recall@1000.

## Citation

```bibtex
@misc{bonifacio2021mmarco,
  title={mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset},
  author={Bonifacio et al. (2021)},
  year={2021},
  note={arXiv:2108.13897}
}
```

- arXiv: 2108.13897