Mrtydi Eval

---
name: mrtydi-eval
description: Evaluates mono-lingual dense retrieval models across eleven typologically diverse languages by measuring their ability to rank relevant Wikipedia passages for given questions. It probes zero-shot cross-lingual generalization and the effectiveness of sparse-dense hybrid retrieval compared to strong sparse baselines. Use when the user wants to benchmark on Mr. TYDI v1.1, or asks about evaluating this task. Reports MRR@100.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.08787
  bibtex_key: zhang2021mrtydi
  confidence: high
---

# mrtydi-eval

> Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval — Zhang et al. (2021) (arXiv:2108.08787, 2021)

## What this evaluates

Evaluates mono-lingual dense retrieval models across eleven typologically diverse languages by measuring their ability to rank relevant Wikipedia passages for given questions. It probes zero-shot cross-lingual generalization and the effectiveness of sparse-dense hybrid retrieval compared to strong sparse baselines.

## Datasets

- **Mr. TYDI v1.1** — total ?; splits: test (-1); repo https://github.com/castorini/mr.tydi

## Metrics

- `MRR@100` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank at cutoff 100. For each query, compute 1/rank of the first relevant passage in the top-100 retrieved results. Average across all queries in the test set.
- `Recall@100` — range: [0, 1]
  - Recall at cutoff 100. Proportion of queries for which at least one relevant passage appears in the top-100 retrieved results.

## Input / output format

**Input**: A question (query) in one of eleven languages, to be matched against a full Wikipedia corpus in the same language.

**Output**: A ranked list of passages (each containing a Wikipedia article title and passage text) returned by the retrieval model.

## Scoring recipe

```python
def compute_metrics(retrieved_passages, gold_passages, k=100):
    if not gold_passages: return 0.0, 0.0
    rr = 0.0
    found = False
    for i, pid in enumerate(retrieved_passages[:k]):
        if pid in gold_passages:
            rr = 1.0 / (i + 1)
            found = True
            break
    recall = 1.0 if found else 0.0
    return rr, recall
# Average across all queries in the test set
```

## Common pitfalls

- Absolute metric scores vary significantly across languages due to differences in question phrasing and corpus characteristics, making direct cross-lingual comparison misleading without normalization.
- MRR@100 heavily penalizes queries where no relevant passage is retrieved in the top-100, which can mask a model's strong ranking ability for the subset of queries where it does find relevant results.
- Zero-shot dense retrieval models (e.g., mDPR trained on English NQ) often underperform strong sparse baselines (BM25) in non-English languages, highlighting distribution shift rather than model architecture flaws.

## Evidence (verbatim from paper)

> Table 2 reports results on the test set across all eleven languages; mean reciprocal rank (MRR) in the top table and recall in the bottom table, both at a cutoff of 100 hits; the final column reports the average across all languages.

## Citation

```bibtex
@misc{zhang2021mrtydi,
  title={Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2108.08787}
}
```

- arXiv: 2108.08787