T2 Ragbench Eval

---
name: t2-ragbench-eval
description: Evaluates Retrieval-Augmented Generation (RAG) systems on their ability to retrieve relevant text-and-table contexts from financial reports and perform numerical reasoning to answer questions. It measures both retrieval effectiveness and the accuracy of the generated numerical answers. Use when the user wants to benchmark on T2-RAGBench, or asks about evaluating this task. Reports Number Match (NM), MRR@3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12071
  bibtex_key: strich2025t2ragbench
  confidence: high
---

# t2-ragbench-eval

> T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation — Strich et al. (2025) (arXiv:2506.12071, 2025)

## What this evaluates

Evaluates Retrieval-Augmented Generation (RAG) systems on their ability to retrieve relevant text-and-table contexts from financial reports and perform numerical reasoning to answer questions. It measures both retrieval effectiveness and the accuracy of the generated numerical answers.

## Datasets

- **T2-RAGBench** — total 32908; splits: test (-1)

## Metrics

- `Number Match (NM)` **(primary)** — range: percent
  - Percentage of questions where the predicted numerical value matches the gold answer within a relative tolerance of epsilon=1e-2. Non-numeric or out-of-tolerance predictions are scored as 0.
- `MRR@3` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank of the first relevant document within the top-3 retrieved results. Calculated as the average of 1/rank for each query.

## Input / output format

**Input**: A question and the top-3 retrieved documents (in markdown format) passed to the generator.

**Output**: A numerical answer or text containing the predicted value.

## Scoring recipe

```python
def score_nm(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            if abs(float(pred) - float(gold)) / abs(float(gold)) <= 1e-2:
                correct += 1
        except:
            pass
    return correct / len(golds)

def score_mrr(retrieval_ranks, k=3):
    scores = []
    for rank in retrieval_ranks:
        if rank <= k:
            scores.append(1.0 / rank)
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Limiting retrieval to top-3 documents is critical; using more degrades LLM performance and is deemed impractical.
- Summarization-based augmentation often drops Number Match because essential numerical details are lost during condensation.
- Cross-encoder rerankers trained on text-only corpora underperform on text-and-table data compared to hybrid BM25.

## Evidence (verbatim from paper)

> We use Number Match and MRR@k as our main metrics as defined in Section[3], but also report Recall@1 (R@1) and Recall@3 (R@3) in the Appendix[I] for better comparability and transparency. Number Match evaluates if a numerical prediction closely matches the gold numerical answer. It compares predicted and ground truth values using relative tolerance ($\epsilon\=1\mathrm{e}{-2}$), accounting for scale invariance. Non-numeric predictions or mismatches are considered incorrect. For MRR we choose $k\=3$, what measures whether the first relevant document appears in the top-3 retrieved results, rewarding higher ranks.

## Citation

```bibtex
@misc{strich2025t2ragbench,
  title={T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation},
  author={Strich et al. (2025)},
  year={2025},
  note={arXiv:2506.12071}
}
```

- arXiv: 2506.12071