Mteb Retrieval Eval

---
name: mteb-retrieval-eval
description: Evaluates text embedding models on information retrieval tasks using the MTEB benchmark and a custom e-commerce Q&A dataset. It measures ranking quality via nDCG and mAP, and assesses similarity distribution calibration via AUPRC on a held-out set with single-relevant-passage queries. Use when the user wants to benchmark on MTEB Retrieval, E-commerce Q&A, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.11868
  bibtex_key: lu2024improvingembedding
  confidence: high
---

# mteb-retrieval-eval

> Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores — Lu et al. (2024) (arXiv:2408.11868, 2024)

## What this evaluates

Evaluates text embedding models on information retrieval tasks using the MTEB benchmark and a custom e-commerce Q&A dataset. It measures ranking quality via nDCG and mAP, and assesses similarity distribution calibration via AUPRC on a held-out set with single-relevant-passage queries.

## Datasets

- **MTEB Retrieval** — total ?; splits: test (-1)
- **E-commerce Q&A** — total 26; splits: train (1040), test (546)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computed as DCG@10 divided by the ideal DCG@10 for the same set of labels. DCG sums (2^rel_i - 1) / log2(i+2) for the top-10 results.
- `mAP@10` — range: [0, 1]
  - Mean Average Precision at rank 10. Computes the average precision score for each query over the top-10 retrieved results, then averages across all queries.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve. Calculated by varying a similarity threshold to generate precision-recall pairs and integrating the curve.

## Input / output format

**Input**: Query text and a list of candidate passage texts.

**Output**: Similarity scores for each query-passage pair, used to rank passages.

## Scoring recipe

```python
def compute_ndcg_at_k(scores, labels, k=10):
    dcg = sum((2**labels[i] - 1) / math.log2(i + 2) for i in range(min(k, len(scores))))
    ideal = sorted(labels, reverse=True)
    idcg = sum((2**ideal[i] - 1) / math.log2(i + 2) for i in range(min(k, len(ideal))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_mAP_at_k(scores, labels, k=10):
    ranked = sorted(zip(scores, labels), reverse=True)[:k]
    rel_count, p_sum = 0, 0.0
    for i, (_, l) in enumerate(ranked):
        if l == 1:
            rel_count += 1
            p_sum += rel_count / (i + 1)
    return p_sum / max(rel_count, 1)

def compute_auprc(scores, labels):
    thresholds = sorted(set(scores), reverse=True)
    precs, recs = [], []
    for t in thresholds:
        tp = sum(1 for s, l in zip(scores, labels) if s >= t and l == 1)
        fp = sum(1 for s, l in zip(scores, labels) if s >= t and l == 0)
        fn = sum(1 for s, l in zip(scores, labels) if s < t and l == 1)
        precs.append(tp / (tp + fp) if (tp + fp) > 0 else 0.0)
        recs.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return np.trapz(precs, recs)
```

## Common pitfalls

- The custom held-out dataset has exactly one relevant passage per query, making standard MTEB retrieval metrics unsuitable; AUPRC is used instead.
- Hard-label fine-tuning on this small dataset actually degrades performance compared to the benchmark, contrary to expectations that more data/labels always help.
- MTEB includes non-retrieval tasks (e.g., bitext mining), but the evaluation strictly focuses on the retrieval subsets.

## Evidence (verbatim from paper)

> The evaluation metrics used are mean average precision at 10 (mAP@10), normalized discounted cumulative gain at 10 (nDCG@10), mean reciprocal rank at 10 (mRR@10), and more (Muennighoff et al., [2022]). In this work, we focus on the retrieval subsets. For retrieval tasks, the default metric suggested by MTEB is nDCG@10, while we show both mAP@10 and nDCG@10 metrics for evaluation. ... The metric of the area under precision-recall curve (AUPRC) is considered.

## Citation

```bibtex
@misc{lu2024improvingembedding,
  title={Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2408.11868}
}
```

- arXiv: 2408.11868