Rar B Eval

---
name: rar-b-eval
description: Evaluates whether dense retrievers and re-rankers can semantically encode and retrieve correct answers to reasoning problems across diverse tasks. It probes the retriever-LLM behavioral gap by testing performance with and without task instructions, and compares full-dataset retrieval against multiple-choice retrieval settings. Use when the user wants to benchmark on RAR-b, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.06347
  bibtex_key: xiao2024rarb
  confidence: high
---

# rar-b-eval

> RAR-b: Reasoning as Retrieval Benchmark — Xiao et al. (2024) (arXiv:2404.06347, 2024)

## What this evaluates

Evaluates whether dense retrievers and re-rankers can semantically encode and retrieve correct answers to reasoning problems across diverse tasks. It probes the retriever-LLM behavioral gap by testing performance with and without task instructions, and compares full-dataset retrieval against multiple-choice retrieval settings.

## Datasets

- **RAR-b** — total ?; splits: test (-1); repo https://github.com/gowitheflow-1998/RAR-b

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computed per task and then averaged using the geometric mean across tasks: G = (∏_{i=1}^{n} x_i)^{1/n}, where n is the number of tasks and x_i is the performance on task i.

## Input / output format

**Input**: A reasoning question or query, optionally prepended with a task instruction. The model retrieves from a candidate pool consisting of either all available answers/documents or a multiple-choice set.

**Output**: A ranked list of candidate answers/documents. Performance is evaluated based on the rank of the ground-truth answer within the top-10 results.

## Scoring recipe

```python
def compute_ndcg_at_10(retrieved, gold, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(retrieved[:k]) if doc == gold)
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

def evaluate_rar_b(predictions, golds, tasks):
    task_scores = []
    for task in tasks:
        scores = [compute_ndcg_at_10(p, g) for p, g in zip(predictions[task], golds[task])]
        task_scores.append(sum(scores) / len(scores))
    return math.exp(sum(math.log(s) for s in task_scores) / len(task_scores))
```

## Common pitfalls

- Using arithmetic mean instead of geometric mean to average performance across tasks, which biases results toward easier tasks due to vastly different performance scales.
- Ignoring the instruction condition: models not trained for instruction-following degrade when instructions are prepended, masking true retrieval capability.
- Full-dataset retrieval can be gamed by simple entity/keyword matching (e.g., in Winogrande), whereas the Multiple-choice Retrieval (MCR) setting is required to reveal nuanced understanding.

## Evidence (verbatim from paper)

> Table[2] presents the results for nDCG@10 performance. Because of the different scales of nDCG@10 across tasks due to different task difficulties and corpus sizes, we take the geometric mean across tasks to represent each model’s average performance, which is given by $G=\left(\prod_{i\=1}^{n}x_{i}\right)^{\frac{1}{n}}$, where $n$ is the number of tasks and $x_{i}$ represent the performance of each task.

## Citation

```bibtex
@misc{xiao2024rarb,
  title={RAR-b: Reasoning as Retrieval Benchmark},
  author={Xiao et al. (2024)},
  year={2024},
  note={arXiv:2404.06347}
}
```

- arXiv: 2404.06347