Miracl Eval

---
name: miracl-eval
description: Evaluates multi-lingual ad-hoc retrieval across 18 languages, testing a model's ability to match queries and passages in the same language using dense, sparse, and multi-vector embedding strategies. Use when the user wants to benchmark on MIRACL, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.03216
  bibtex_key: chen2024m3embedding
  confidence: high
---

# miracl-eval

> M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — Chen et al. (2024) (arXiv:2402.03216, 2024)

## What this evaluates

Evaluates multi-lingual ad-hoc retrieval across 18 languages, testing a model's ability to match queries and passages in the same language using dense, sparse, and multi-vector embedding strategies.

## Datasets

- **MIRACL** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list of retrieved passages relative to the ideal ranking, normalized by the ideal DCG@10.
- `Recall@100` — range: [0, 1]
  - The proportion of relevant passages found within the top 100 retrieved results.

## Input / output format

**Input**: A query and a corpus of passages, all presented in the same language.

**Output**: A ranked list of top-1000 candidates (for dense/sparse) or top-200 candidates (for multi-vector reranking), combined via weighted fusion for hybrid methods.

## Scoring recipe

```python
def compute_ndcg_at_10(retrieved_ids, relevant_ids):
    dcg = 0.0
    for i, doc_id in enumerate(retrieved_ids[:10]):
        if doc_id in relevant_ids:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), 10)))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- The paper uses different retrieval pipelines for dense/sparse (Faiss/Lucene top-1000) vs multi-vector (reranker top-200), which affects latency and recall.
- Hybrid retrieval weights (w1, w2, w3) are fixed per task (e.g., w1=1, w2=0.3 for MIRACL) rather than tuned per language.

## Evidence (verbatim from paper)

> We evaluate the multi-lingual retrieval performance with MIRACL, which consists of ad-hoc retrieval tasks in 18 languages. Each task is made up of query and passage presented in the same language. Following the official benchmark, we evaluate our method using Pyserini, and use nDCG@10 as the primary evaluation metric (Recall@100 is also measured and reported in Appendix[C.1]).

## Citation

```bibtex
@misc{chen2024m3embedding,
  title={M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2402.03216}
}
```

- arXiv: 2402.03216