Beir Eval

---
name: beir-eval
description: Evaluates zero-shot information retrieval capabilities across 18 diverse domains and query types. It probes a model's ability to retrieve relevant documents without domain-specific fine-tuning, highlighting performance variations due to domain shifts, query length, and lexical versus semantic matching. Use when the user wants to benchmark on BEIR, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.07471
  bibtex_key: kamalloo2023beir
  confidence: high
---

# beir-eval

> Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard — Kamalloo et al. (2023) (arXiv:2306.07471, 2023)

## What this evaluates

Evaluates zero-shot information retrieval capabilities across 18 diverse domains and query types. It probes a model's ability to retrieve relevant documents without domain-specific fine-tuning, highlighting performance variations due to domain shifts, query length, and lexical versus semantic matching.

## Datasets

- **BEIR** — total ?; splits: test (-1); HF `beir/benchmark`; repo https://github.com/beir-cellar/beir

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved documents, discounted logarithmically by position, and normalized by the ideal DCG for that query.
- `recall@100` — range: [0, 1]
  - Recall at rank 100. Calculates the fraction of all relevant documents in the corpus that appear within the top 100 retrieved results.

## Input / output format

**Input**: Query text and a target corpus of documents. Models receive queries and must rank documents from the corpus.

**Output**: Ranked list of document IDs or passages. Results are typically reported for top-10 and top-100 ranks.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    ndcg_vals, rec_vals = [], []
    for qid, ranked in predictions.items():
        rel_set = gold[qid]
        dcg = sum(1.0 / log2(i + 2) for i, doc in enumerate(ranked[:10]) if doc in rel_set)
        idcg = sum(1.0 / log2(i + 2) for i in range(min(10, len(rel_set))))
        ndcg_vals.append(dcg / idcg if idcg > 0 else 0.0)
        hits = sum(1 for doc in ranked[:100] if doc in rel_set)
        rec_vals.append(hits / len(rel_set) if rel_set else 0.0)
    return {'nDCG@10': mean(ndcg_vals), 'recall@100': mean(rec_vals)}
```

## Common pitfalls

- Macro-averaging scores across all 18 BEIR datasets can obscure severe performance drops on specific domains (e.g., biomedical or short-query tasks).
- Relevance judgments in some datasets (e.g., Toulouse-2020) are shallow and biased toward lexical matches, unfairly penalizing semantic retrieval models.
- Zero-shot evaluation means no domain-specific fine-tuning is allowed, so results heavily reflect pre-training data overlap rather than true generalization.

## Evidence (verbatim from paper)

> The effectiveness of the five models presented in the previous section is shown in Table 2, with nDCG@10 in the left group of columns and recall@100 in the right group of columns. Each row corresponds to one of the BEIR datasets, and the rows are ordered in the same manner as Thakur et al. [34].

## Citation

```bibtex
@misc{kamalloo2023beir,
  title={Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard},
  author={Kamalloo et al. (2023)},
  year={2023},
  note={arXiv:2306.07471}
}
```

- arXiv: 2306.07471