Granite R2 Retrieval Eval

---
name: granite-r2-retrieval-eval
description: This evaluation protocol assesses the retrieval and reranking capabilities of encoder-based embedding models across diverse domains including general text, code, long documents, tables, and multi-turn conversations. It also measures encoding speed to evaluate efficiency in large-scale document ingestion pipelines. Use when the user wants to benchmark on MTEB-v2, BEIR, COIR, MLDR, LongEmbed, Table IR, MT-RAG, IBM Documentation, Miracl, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.21085
  bibtex_key: awasthy2025graniteembeddingr2
  confidence: high
---

# granite-r2-retrieval-eval

> Granite Embedding R2 Models — Awasthy et al. (2025) (arXiv:2508.21085, 2025)

## What this evaluates

This evaluation protocol assesses the retrieval and reranking capabilities of encoder-based embedding models across diverse domains including general text, code, long documents, tables, and multi-turn conversations. It also measures encoding speed to evaluate efficiency in large-scale document ingestion pipelines.

## Datasets

- **MTEB-v2** — total ?; splits: test (-1)
- **BEIR** — total ?; splits: test (-1)
- **COIR** — total ?; splits: test (-1)
- **MLDR** — total ?; splits: test (-1)
- **LongEmbed** — total ?; splits: test (-1)
- **Table IR** — total ?; splits: test (-1)
- **MT-RAG** — total ?; splits: test (-1)
- **IBM Documentation** — total 23000; splits: test (23000)
- **Miracl** — total ?; splits: test (-1)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores up to position 10, normalized by the ideal DCG@10. Standard convention for ranking retrieval tasks.
- `Recall@5` — range: [0, 1]
  - Fraction of relevant documents retrieved within the top 5 results. Used for MTRAG and specific Table IR tasks.
- `Match@5` — range: [0, 1]
  - Exact match rate for the top 5 retrieved documents. Used for two Table IR tasks.
- `Accuracy@1` — range: [0, 1]
  - Proportion of queries where the single top-ranked document is correct. Used for two LongEmbed datasets.
- `Encoding Speed (Docs/s)` — range: other
  - Number of documents processed per second on a single Nvidia H100 GPU with batch size 128. Documents are chunked to 512 tokens with 100-token overlap.

## Input / output format

**Input**: Query and document pairs (or chunks). For retrieval, max sequence length is 8192 tokens. For speed benchmark, documents are chunked into 512 tokens with 100-token overlap. For reranking, queries are truncated to 64 tokens and top-20 retrieved documents are provided.

**Output**: Dense embedding vectors (768-dim for base models, 384-dim for small models) or reranking scores/rankings for the candidate documents.

## Scoring recipe

```python
def compute_avg_metric(predictions, gold, dataset_name):
    if dataset_name in ['MTRAG', 'NQTables', 'OTT-QA', 'MultiHierTT']:
        return recall_at_k(predictions, gold, k=5)
    elif dataset_name in ['AIT-QA']:
        return match_at_k(predictions, gold, k=5)
    elif dataset_name in ['LongEmbed_subset_1', 'LongEmbed_subset_2']:
        return accuracy_at_k(predictions, gold, k=1)
    else:
        return ndcg_at_k(predictions, gold, k=10)

def benchmark_score(predictions_map, gold_map, benchmark):
    scores = []
    for task in benchmark.tasks:
        scores.append(compute_avg_metric(predictions_map[task], gold_map[task], task))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Different benchmarks use different primary metrics (NDCG@10, Recall@5, Match@5, Accuracy@1); averaging requires mapping each dataset to its correct metric.
- Speed evaluation uses a fixed 512-token chunk size with 100-token overlap on a single H100 GPU with batch size 128, which may not generalize to other hardware or chunking strategies.
- Reranker evaluation is conditioned on top-20 documents retrieved by specific Granite embedding models, not a standard open retriever, limiting direct comparison with other rerankers.

## Evidence (verbatim from paper)

> Table 2: Retrieval Performance. Average scores are reported for benchmarks, with the number of tasks indicated in parentheses. MTRAG shows Recall@5. Two datasets in LongEmbed use Accuracy@1. Three tasks in Table IR use Recall@5 and two use Match@5. All other scores are average NDCG@10. Complete breakdown of scores is provided in Appendix [D].

## Citation

```bibtex
@misc{awasthy2025graniteembeddingr2,
  title={Granite Embedding R2 Models},
  author={Awasthy et al. (2025)},
  year={2025},
  note={arXiv:2508.21085}
}
```

- arXiv: 2508.21085