Mair Eval

---
name: mair-eval
description: Evaluates retrieval models' ability to follow complex, task-specific instructions across diverse domains and long-tail tasks. It measures how instruction tuning impacts generalization and performance on heterogeneous query-document relevance tasks compared to non-instruction-tuned baselines. Use when the user wants to benchmark on MAIR, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.10127
  bibtex_key: sun2024mair
  confidence: high
---

# mair-eval

> MAIR: A Massive Benchmark for Evaluating Instructed Retrieval — Sun et al. (2024) (arXiv:2410.10127, 2024)

## What this evaluates

Evaluates retrieval models' ability to follow complex, task-specific instructions across diverse domains and long-tail tasks. It measures how instruction tuning impacts generalization and performance on heterogeneous query-document relevance tasks compared to non-instruction-tuned baselines.

## Datasets

- **MAIR** — total ?; splits: test (-1); repo https://github.com/sunnweiwei/Mair

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Calculated as the ratio of the DCG@10 of the predicted ranking to the ideal DCG@10. The overall score is the average across all queries, with domain-level averages also reported.

## Input / output format

**Input**: For embedding models: a query paired with a task-specific instruction. For re-rankers: a query, instruction, and a candidate passage (from top-100 retrieved by a first-stage retriever). Passages are truncated to the model's maximum input length.

**Output**: A ranked list of passages (top-100 for re-rankers, evaluated at top-10).

## Scoring recipe

```python
import math
def compute_ndcg_at_10(gold_relevance, predicted_ranking):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:10]):
        rel = gold_relevance.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold_relevance.values(), reverse=True)[:10]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Non-instruction-tuned models are evaluated under the '+ instruction' setting for reference, which may overstate their actual instruction-following capability.
- Re-ranking models rely on a first-stage retriever (text-embedding-3-small) to fetch the top-100 candidates, so their reported performance conflates retrieval and re-ranking quality.
- Passages are truncated to each model's maximum input length, which can degrade relevance scoring for long documents and affect nDCG@10 scores.

## Evidence (verbatim from paper)

> Following previous work, we use nDCG@10 as the evaluation metric. The overall score is defined as the average score across all queries. We also report the average nDCG@10 for each of the following domains: Web, Academic, Code, Medical, Legal, and Finance.

## Citation

```bibtex
@misc{sun2024mair,
  title={MAIR: A Massive Benchmark for Evaluating Instructed Retrieval},
  author={Sun et al. (2024)},
  year={2024},
  note={arXiv:2410.10127}
}
```

- arXiv: 2410.10127