Messirve Eval

---
name: messirve-eval
description: Evaluates information retrieval models on a large-scale, dialectally diverse Spanish dataset. It probes the ability of lexical and dense retrieval models to rank relevant Wikipedia documents for real-world Spanish search queries without fine-tuning. Use when the user wants to benchmark on MessIRve, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.05994
  bibtex_key: valentini2024messirve
  confidence: high
---

# messirve-eval

> MessIRve: A Large-Scale Spanish Information Retrieval Dataset — Valentini et al. (2024) (arXiv:2409.05994, 2024)

## What this evaluates

Evaluates information retrieval models on a large-scale, dialectally diverse Spanish dataset. It probes the ability of lexical and dense retrieval models to rank relevant Wikipedia documents for real-world Spanish search queries without fine-tuning.

## Datasets

- **MessIRve** — total ?; splits: test (-1)

## Metrics

- `Recall@100` — range: [0, 1]
  - The fraction of relevant documents within the top 100 results, averaged over all queries.
- `nDCG@10` **(primary)** — range: [0, 1]
  - The normalized Discounted Cumulative Gain. It compares the rank of the top 10 results to the ideal ranking where relevant documents are ranked higher. It is averaged over all queries.

## Input / output format

**Input**: Query string and a corpus of Wikipedia documents (with titles appended to document text before retrieval).

**Output**: Ranked list of document IDs/URLs for each query.

## Scoring recipe

```python
def recall_at_k(retrieved, relevant, k=100):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)

def ndcg_at_k(retrieved, rel_scores, k=10):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rel_scores[:k]))
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(sorted(rel_scores, reverse=True)[:k]))
    return dcg / idcg if idcg > 0 else 0.0

# Average over all queries
recall_avg = mean(recall_at_k(q_ret, q_rel) for q in queries)
ndcg_avg = mean(ndcg_at_k(q_ret, q_scores) for q in queries)
```

## Common pitfalls

- Models are evaluated in a strict zero-shot setting; fine-tuning on the dataset is explicitly excluded.
- Wikipedia article titles are appended to document text before retrieval, which is a specific preprocessing step that affects baseline performance.
- Evaluation scores are averaged over all queries, though per-dialect partitions (e.g., ar, mx, bo, gt) are also reported.

## Evidence (verbatim from paper)

> We use two standard metrics to measure retrieval performance: Recall@100: the fraction of relevant documents within the top 100 results, averaged over all queries. nDCG@10: the normalized Discounted Cumulative Gain. It compares the rank of the top 10 results to the ideal ranking where relevant documents are ranked higher. It is averaged over all queries.

## Citation

```bibtex
@misc{valentini2024messirve,
  title={MessIRve: A Large-Scale Spanish Information Retrieval Dataset},
  author={Valentini et al. (2024)},
  year={2024},
  note={arXiv:2409.05994}
}
```

- arXiv: 2409.05994