Mt Raig Eval

---
name: mt-raig-eval
description: This benchmark evaluates retrieval-augmented insight generation over multiple tables. It requires models to retrieve relevant tables from a database and synthesize multi-hop insights across them, assessing both faithfulness and completeness of the generated reasoning. Use when the user wants to benchmark on MT-RAIG Bench, or asks about evaluating this task. Reports MT-RAIG Eval.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.11735
  bibtex_key: seo2025mt_raig
  confidence: high
---

# mt-raig-eval

> MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables — Kwangwook Seo, Donguk Kwon, Dongha Lee (2025) (arXiv:2502.11735, 2025)

## What this evaluates

This benchmark evaluates retrieval-augmented insight generation over multiple tables. It requires models to retrieve relevant tables from a database and synthesize multi-hop insights across them, assessing both faithfulness and completeness of the generated reasoning.

## Datasets

- **MT-RAIG Bench** — total ?; splits: (unstated)

## Metrics

- `MT-RAIG Eval` **(primary)** — range: percent
  - Decomposes generated insights into table-aware and question-aware components. Auto-evaluation scores are normalized to a human preference scale of [-1, 0, 1], and the metric reports the Pearson correlation with human preference ratings.

## Input / output format

**Input**: Natural language query/question and a database of tables for retrieval; for generation, the query and retrieved tables.

**Output**: A synthesized multi-hop insight text that answers the query using information from multiple tables.

## Scoring recipe

```python
def mt_raig_eval(predictions, golds, human_preferences):
    # Normalize auto-evaluation score differences to [-1, 0, 1]
    normalized_scores = normalize_to_preference_scale(predictions, golds)
    # Compute Pearson correlation with human ratings
    faithfulness_corr = pearson_correlation(normalized_scores['faithfulness'], human_preferences['faithfulness'])
    completeness_corr = pearson_correlation(normalized_scores['completeness'], human_preferences['completeness'])
    return faithfulness_corr, completeness_corr
```

## Common pitfalls

- Table-specific embedding models underperform general text-based embeddings for this task.
- Increasing the number of retrieved tables (k) beyond a certain threshold introduces noise, causing faithfulness to drop while completeness remains stable.
- Existing TQA methods specialize in single-table factoid extraction and struggle with multi-table reasoning compared to general-purpose LLMs.

## Evidence (verbatim from paper)

> For scoring, we follow the setting of Ru et al. ([2024]) to normalize each auto-evaluation score difference to the human preference scale of [-1,0,1] and then measure the Pearson correlation with human preference ratings. Additionally, we report the correlation between the human evaluators as the upper bound. From the results in Table[5], we can observe that MT-RAIG Eval achieves the highest correlation with human preference ratings across both dimensions, demonstrating its reliability over baseline methods for evaluating MT-RAIG Bench.

## Citation

```bibtex
@misc{seo2025mt_raig,
  title={MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables},
  author={Kwangwook Seo, Donguk Kwon, Dongha Lee (2025)},
  year={2025},
  note={arXiv:2502.11735}
}
```

- arXiv: 2502.11735