Iclerb Eval

---
name: iclerb-eval
description: Evaluates embedding models and rerankers on their ability to retrieve contextually useful documents for In-Context Learning (ICL) tasks. It measures retrieval effectiveness by ranking candidate documents based on their utility in improving downstream LLM accuracy, rather than relying solely on semantic similarity. Use when the user wants to benchmark on TruthfulQA, Emotion, ProductER, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.18947
  bibtex_key: alghossein2024iclerb
  confidence: high
---

# iclerb-eval

> ICLERB: In-Context Learning Embedding and Reranker Benchmark — Al Ghossein et al. (2024) (arXiv:2411.18947, 2024)

## What this evaluates

Evaluates embedding models and rerankers on their ability to retrieve contextually useful documents for In-Context Learning (ICL) tasks. It measures retrieval effectiveness by ranking candidate documents based on their utility in improving downstream LLM accuracy, rather than relying solely on semantic similarity.

## Datasets

- **TruthfulQA** — total ?; splits: test (-1)
- **Emotion** — total ?; splits: test (-1)
- **ProductER** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by applying a logarithmic discount to the relevance score of each retrieved document, normalized by the ideal DCG@10. Values range from 0 to 1, where 1 indicates a perfect ranking.

## Input / output format

**Input**: A query representing an ICL task or prompt, paired with a pool of candidate documents or context snippets to be retrieved and ranked.

**Output**: A ranked list of candidate documents, typically evaluated on the top-10 results.

## Scoring recipe

```python
def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:k]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Rerankers fine-tuned for general semantic similarity or web search may underperform on ICL tasks compared to embedding models.
- Larger model size does not guarantee better ICL retrieval performance; smaller models can significantly outperform larger counterparts.

## Evidence (verbatim from paper)

> Table 2 presents the ICLERB results, sorted by nDCG@10. The top performers on the benchmark are cm-rerank-mxbai-rlaif-v0.1 (our proposed model, detailed in Section 5), BAAI's bge-en-icl, and NVIDIA's NV-Embed-v2.

## Citation

```bibtex
@misc{alghossein2024iclerb,
  title={ICLERB: In-Context Learning Embedding and Reranker Benchmark},
  author={Al Ghossein et al. (2024)},
  year={2024},
  note={arXiv:2411.18947}
}
```

- arXiv: 2411.18947