Swim Ir Eval

---
name: swim-ir-eval
description: Evaluates multilingual dense retrieval models on cross-lingual and monolingual open retrieval tasks. It measures how effectively synthetic LLM-generated training data scales retrieval performance compared to human-labeled baselines across diverse languages and corpus sizes. Use when the user wants to benchmark on XOR-Retrieve, MIRACL, XTREME-UP, or asks about evaluating this task. Reports Recall@mkt.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.05800
  bibtex_key: thakur2023swimir
  confidence: high
---

# swim-ir-eval

> Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval — Thakur et al. (2023) (arXiv:2311.05800, 2023)

## What this evaluates

Evaluates multilingual dense retrieval models on cross-lingual and monolingual open retrieval tasks. It measures how effectively synthetic LLM-generated training data scales retrieval performance compared to human-labeled baselines across diverse languages and corpus sizes.

## Datasets

- **XOR-Retrieve** — total 17000; splits: train (15000), dev (2000)
- **MIRACL** — total 101783; splits: train (88288), dev (13495)
- **XTREME-UP** — total 15985; splits: train (5280), test (10705)

## Metrics

- `Recall@mkt` **(primary)** — range: [0, 1]
  - Computes the fraction of queries for which the minimal answer is contained within the top m thousand tokens of the retrieved passages. Evaluated at m=5 and m=2.
- `nDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality based on graded relevance judgments.
- `Recall@100` — range: [0, 1]
  - Fraction of queries where the relevant passage appears in the top 100 retrieved results.
- `MRR@10` — range: [0, 1]
  - Mean Reciprocal Rank at rank 10, averaging the inverse rank of the first relevant passage for each query.

## Input / output format

**Input**: Query string (monolingual or multilingual) and a passage corpus (Wikipedia or TyDi-QA passages).

**Output**: Ranked list of passages (or top-k passages) for each query.

## Scoring recipe

```python
def compute_recall_mkt(retrieved_passages, gold_answer, k=5):
    token_limit = k * 1000
    acc_tokens = 0
    for p in retrieved_passages:
        acc_tokens += len(p.split())
        if acc_tokens >= token_limit:
            break
    return 1.0 if gold_answer in retrieved_passages else 0.0
```

## Common pitfalls

- SWIM-X models were not optimized with hard negatives, unlike supervised baselines that use up to four hard negatives per pair.
- Corpus token limits vary by dataset (e.g., 100-word tokens for XOR-Retrieve, varying sizes for MIRACL), affecting Recall@mkt calculations.
- Translation script mismatches (e.g., Manipuri in 'Meitei' vs 'Bengali-Assamese' script) can cause evaluation failures if not handled.

## Evidence (verbatim from paper)

> We evaluate our models using recall at m kilo-tokens, i.e., Recall@mkt, which computes the fraction of queries for which the minimal answer is contained within the top m thousand tokens of the retrieved passages. Following prior work in Asai et al. (2021a), we evaluate our models at Recall@5kt and Recall@2kt.

## Citation

```bibtex
@misc{thakur2023swimir,
  title={Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval},
  author={Thakur et al. (2023)},
  year={2023},
  note={arXiv:2311.05800}
}
```

- arXiv: 2311.05800